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The Future of Text Journal 


Welcome to The Future Text Lab’s Journal, of which this is the last issue produced before 
Apple ‘unleashed’ the Reality Pro VR headset, as it is expected to be called, published 4th of 
June 2023. 

Our intention with everything we do is to promote the thinking that even though text 
has been critical to expanding human thought for millennia, with digital text we can go much 
further than we have done so far, with AI further still and with VR even further. If we do not 
augment how we interact with our knowledge through text, we risk not only being 


overwhelmed but also mis-informed, particularly with the threat posed by AI. 


If you download the other issues from https://futuretextpublishing.com and if you use our 
‘Reader’ PDF viewer for macOS (downloadt), for macOS, you can interact with this 
document in ways you cannot with a regular PDF viewer, because this document features a 
Visual-Meat appendix: https://visual-meta.info. Please keep in mind that this level of 
metadata can also make rooms in VR environments possible, something we hope to be able 


to experiment with: 


Keywords & Views 


The list of keywords on the page opposite are completely arbitrarily chosen by the editor as 
an experiment to help you go through this document in a more non-linear way. 

If you are reading this document in a traditional PDF viewer, such as in Apple’s 
‘Preview’, you can select any of the keywords here, copy it, do #-F to invoke the ‘Find’ 
command, then paste it and this will result in all the occurrences of that keyword appearing in 
a column on the left of your document. 

If you use Adobe ‘Acrobat’ you can also copy and paste into the Find dialog and hit 
‘enter’. 

If you are using the Augmented Text ‘Reader’, which was developed in conjunction 
with the Future Text Lab for our Books and Journal, there are two distinct ways you can use 
the keywords. The first is similar to how you would use Preview or Acrobat, but simpler: 
Select a keyword and do 3 -F and you will instantly get your result, there is no need to copy 
and paste into a dialog. Your view of the document will instantly change to show you only the 
lines in the document which contains your selected text, plus one grey line above and below 
for context. You can now click on an occurrence to jump to its location or you can do #-F 
again to exit this view, since in many cases all you needed from the Find was in this view. 


Here are suggested keywords: 


e AR, VR, XR, MR, xrOS, RealityPro, Quest, HMD, headset, metaverse, cyberspace 

e Intelligence Augmentation, IA, sensemaking, knowledge work, tools for thought, meeting, 
collaboration, multidimensional, knowledge map, graph, map 

e AI, Artificial Intelligence, speech recognition, natural language processing, NLP, 
computer vision, machine learning, AGI, Artificial general intelligence, OpenAI, 
DeepMind, Anthropic, LaAMDA, Language Model for Dialogue Applications, Bard, 
ChatGPT, chatbot, bot, reinforcement learning, RL, Softspace, large language model, 
LLM 

e architecture, museum, exhibitions, structure, room, building, CAD, world, space 

e Meta, Facebook, Microsoft, Google, Softspace, BigTech, Scholarcy, Augmented Text 
Company, Future Text Publishing, Future Text Lab 

e Art, artist, poet, poetry, art, painting, sculpture, writer 

e Human, person, people, society, culture, dream, creativity, ethics 


e MacOS, iOS, Android, Windows 


A more colourful approach to keywords. 


If you are using Reader 7.0 or above, you don’t need to interact with the keywords 
individually, as further described in the Coda for this Journal issue. As you can see, they are 
in the normal body font and they are black, but if you do #-shift-K you will see only the 
keywords. Additionally, if you fold the document into an outline, you will see the keywords 
appear under any heading where that section contains that keyword, in colour. This is an 
option you can choose to turn off in Reader’s Settings, where you can also edit these 
keywords to suit your needs. To be clear, the keywords printed after this section are not the 


reason for the colours, the ones in the Settings are. 


The default colour sets are tuned to this Journal since it an experimental feature in 


support of the work we are doing the the Future Text Lab. 


The idea is not that the colours are going to necessarily be very readable in this view 
and it would be a little presumptuous of us to think you will want to take the time to learn 
what each colour corresponds to, which is why the colours are only colours and they are not 
listed by labels. What the goal of the colours is, is simply that over time, when you see them 
in this folded overview, you will get more of a sense of what each colour represents, not as 
logical one-to-one representation necessarily, but as a more of a feeling. For example, this 
work on the future of text currently includes more discussions on VR and AI than before, and 


in this view the text which refers to VR will be in red whereas AI will be in dark green. 


There are very few colours even those of us who do not have a form of colour blindness 
can distinguish when the colours are not next to each other and there are very few colours 
which are highly legible on either/or light and dark backgrounds, but the aim here is not for 
full legibility nor for one solution to fit all, but rather an experiment for how we can make 
text more visible for overviews without having to ‘literally’ read the text. A point could be 
made for not showing the keywords at all, only dots or lines, but we have taken this approach 
with coloured words since this will, perhaps, over time help you learn what the different 
colours denote. A further note on the colours is that we have chosen for many to be in pairs, 
such as light blue and dark blue and light green and dark green, but also red and light red 
(pink) and orange and dark orange (brown), in order to group categories together a little bit. 
For example, we are experimenting with dark blue for the names of those who have authored 
an article for either a book or Journal issue, and light blue for those members of the 
community who have provided serious input and dialog but have so far chosen not to write an 
article. For similar reasons, architectural terms are in orange since that relates to the red of 


virtual architecture and knowledge work tools and techniques (building mental structures) in 


dark orange/brown. 


Your feedback on this experiment feature would be very much appreciated and please 


feel feel free to play around with other sets of keywords, perhaps using only a few until you 


get used to their meaning. 


In Reader you can also do the following, in addition to #-F for Find, to more flexibly view 


the contents of this Journal: 


d6-shift-N to see only Names and headings 

d6-shift-D to see any Defined terms (terms in this document’s ‘Glossary’) and headings 

Z to zoom in and out of the document one step. Z again to zoom back out 

d6 -shift-+ to zoom in step by step 

Select text and ’S’ to have the text spoken to you 

[ESC] to quickly toggle in and out of full screen two-page view 

Select text and [spacebar] to see the text full screen, single column for easier reading. You 
can do this on a selection of text or the whole document by doing #-A first, though for 


longer documents this is not as convenient as selecting paragraphs of specific text. You 


would like to focus on [spacebar] or click [Exit] to cancel out of this view 


Navigation 


Arrow right for next page or [spacebar] as you might expect — — 

Down and up arrow for next and previous article | f This is a real time saver and will 
allow you to get a view of what this document contains which does not force you to right 
arrow every page which can make it easy to overshoot the next interesting article, but with 
down arrow going to the next article, you can click and see not only the title but also the 
full article, allowing you to get a good impression and choose whether to read the article 
or jump straight to the next article, no matter how many—or few—pages it is after the one 


you are looking at now 


d6-minus to fold the document into an outline and #= to unfold. Unless you have 
changed the setting, this view will fold the document to only show headings and any 


keywords which appear under the headings. 


Annotation 


Select text and ‘H’ to highlight text in grey 

Select text and cmd-shift-H to highlight text in yellow 

‘H’ without selecting text to see all the highlights in the document 

Select text and ‘X’ to add a strikethrough to the text 

If the document contains Visual-Meta (see next page), you can fold the document to see 
headings and all the highlighted text will also appear 


d6-double-click to add an annotation. This results in a clickable yellow box with your 
comments. We hope to improve on this, to at least let the first few words of what you 


wrote be visible even in this little yellow box. 


Citing this Document 


On any of the operations below, if you paste into our ‘Author’ word processor for macOS, the 


citation will be correctly formatted in the body text and added to the References on export. 


d6-C on selected text to Copy the text with full citation information to paste in email, 
word processors and so on. When you paste in Author, the deleted text will appear as a 


quote 


d6 -shift-C on selected text to Copy the text with full citation information to paste in email, 
word processors and so on. In this case, when you paste into Author, the selected text will 
not appear as a quote, but when exported to a new PDF, the user can click on the citation 
in the body text and the selected text will appear in a pop-up, along with a link to the 
original document (if the user has the document on their system, in other words if they 


have read it in Reader before), or a web link to search based on the document’s DOI 


Select text and #€-shift-J to copy text into a Research Journal format to be pasted in 
Author. Once in Author #-J to open the Journal and then -V to paste in a format which 


will make it easier to find the citation in the future. 
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Frode Alexander Hegland : Introduction 


This Journal edition is the last authored before Apple releases their expected VR/AR/XR 
headset which we think will be called ‘Reality Pro’. These are therefore the last words we 
have been able to write down unfiltered by what Apple’s implementation will be like. The 
final rumours are these: “The Worldwide Developers Conference (WWDC), Apple's annual 
developer and software-oriented conference, is now just a few days away. As with previous 
years, Apple is expected to introduce major updates to all of its operating systems, but unique 
to WWDC 2023, Apple is also very likely to unveil xrOS, the software that will run its much- 
rumored AR/VR mixed-reality headset. Here's everything we're expecting from it. Apple has 
not publicly acknowledged the existence of its headset, but internal Apple sources have 
confirmed that the operating system will be called xrOS. xrOS is believed to have an iOS-like 
interface that will be familiar to iPhone and iPad users, but presented to them and interacted 
with in an entirely new way. We are expecting a dedicated Home Screen containing apps and 
customizable widgets, all of which can be rearranged, and all viewed on dual high-resolution 
4K micro OLED displays with up to 3,000 pixels per inch for a rich, realistic, and immersive 
viewing experience. Both of the user's hands and eyes will be tracked by more than a dozen 
optical cameras, and the wearer is expected to be able to look at an on-screen item to select 
it, and use a hand gesture like a pinch to activate the item on the screen. The xrOS software 
could even potentially project AR app interface elements onto actual objects in the real world 
for a mixed-reality overlay kind of effect.” 


macRumors website (archived as a PDF*) 


The step into working in fully immersive technology has started long ago, but I believe that 
Apple has an corporate ‘knack’ of delivering powerful technologies which become welded to 
the user to a degree nobody else has. Witness the iPhone and the Apple Watch, while not 
ignoring the Mac that started it all for Apple. Meta won’t be able to do this because they 
don’t have access to the user’s work data in the same way Apple has so this won’t be a 
‘metaverse’. Microsoft can’t do this because, well I honestly don’t know why, but let’s see 
what happens on the Sth of June 2023, a week from today, when Apple releases their headset. 
A technology which fully envelopes the user has a much higher degree of ownership of 
the user and this matters. The more we are locked in, to companies or technical 
infrastructures, from file formats and network protocols and ways to render scenes, the less 
freedom there is for the ecosystem to evolve. This step is too important for anyone to ‘own’ 


or dominate. 
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At Syracuse University a professor, whose name I do not remember and whom I have 
not been able to track down, said: “Truth Kills Creativity” 


And this highlights how difficult it is to dream. Once the headset is announced, we will 
collectively think about some interfaces and infrastructures that this is ‘just the way it is’. Let 
us never settle for ‘this is just the way it is’. Let us always strive for better. 

The articles contained herein are intended to help you, in the future, get some unbiased 
perspectives on what might have been—what might be. 

This matters. VR (including all flavours, from completely immersive to only slightly 
overlaid our physical reality) will be where we will spend much of our work time so how it 


extends our minds and how it limits our minds will matter. 
As I write in my piece on VR & Al, I believe that VR can truly help us harness the 
power of AI, instead of letting us simply be overwhelmed. 


Soon we will be escaping flatland. 


Powerfully Open 


Dreaming is important, and so is building an evolutionary environment where dreaming 
mattes, where dreams can become real and not remain mere fantasies. This is why most 
powerful thing I wish for for the coming VR, is that we all work to make it as open as 
possible, to enable multiple imaginations to compete. 

The Internet was not a new network, it was a standard for networks to connect, hence 
the name ‘inter’-‘net’. We will need an internet for VR the same for the virtual worlds now 
being unleashed. We need multiple infrastructures for those of us passionately working to 
unleash the augmentation potential of VR to be able to experiment to experience new ways of 
working, thinking and communicating, allowing different approaches to truly compete. 


You are the one actually dealing with this, we were only dreaming. 


Good luck to you! 
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Christopher Gutteridge 


The modern stories around the use of VR seem to be: 


e gaming 
* museums and exhibitions 


e shared meeting and collaboration 


My limited experience of a virtual meeting space was that it entirely missed the power of VR. 
VR as a productivity tool for managing data and documents is generally like simulating all 
the aspects of the real world that make a digital library and desktop better. We add back in the 
walls obscuring things, and the time and effort to navigate. 

In a shared space (or a gaming world) it's useful to make entities behave in ways that 
other observers can understand, but in a personal VR space this isn't needed. It's trivial to 
zoom nearly infinitely, it's possible to move at any speed and rotate the world to any angle. 
This isn't user friendly but neither is blender or vi. I don't want to spend 3 minutes walking 
around something. My time is valuable to me. It's why I learn the keyboard shortcuts. I gain 
nothing from looking at a virtual art gallery that I couldn't get from looking at photographs... 
or at least at a 3D model spinning in a webpage. 

The issue with VR is that it's still the answer to questions people are not asking. It's 
possible it's never going to be the answer to a question, but more likely that we don't ask the 
right questions, or at least not loudly enough. Often to find the "real" application for 
something, we first have the "killer app" -- something that is actually useful and makes it 
worth people engaging with. With Linux it was the GIMP art package and Open Office. For 
the first time you could do common IT tasks on a purely free software stack and that made it 
worth people installing on the desktop. Frankly, VR is yet to have a killer app. I've never 
once had a problem and thought "this would be easier if I had a VR rig". And until it does, 
we're not going to get the unexpected new uses. It's a popular story that SMS was initially 
considered something people would only occasionally use, and that is a formative tech in 
modern youth culture. The 303 synth was designed as a bass guitar replacement <https:// 
www.soundonsound.com/people/history-roland-part-2> which is was crap at, but, again, was 
used in ways the creators never expected when placed in the right (or wrong!) hands. 

A key difference though is that the barrier to finding novel uses for SMS and the 303 was 


very low. VR has a lot more issues... it's been very expensive, it's kinda dorky, and it makes 


13 


the wearer genuinely vulnerable so is only suitable for use somewhere the user feels secure. 


It had a unique chance during the pandemic, when people spent more on 
communication and home entertainment hardware, but it was technology like skype which 
had the massive boost. People wanted to still have some face to face time with coworkers and 
friends. Although after six months my team opted for mostly camera off meetings for internal 
stuff. It wasn't uncommon for someone to be wearing headphones listening to a report while 
putting out the laundry or similar zero-brain chores that the body can be doing while the mind 
is doing the day job. VR is the antithesis of this, it ties you to a location and demands ALL 
your focus. 

It has obviously found some high-end uses like telecontrol of equipment, but that's not 
the killer app that lets 15 year olds innovate with it. 

Lots of world builders exist -- Secondlife and it's imitators, but these feel dangerously close 
to NFT grifts. Selling virtual land to foolish speculators. But what doesn't exist (to my 
knowledge) is tools which let non-experts innovate with the interface itself and the rules of 
the world. Kids should easily be able to define the physics and interactions of a virtual world, 
and share them with each other to remix and build on. That's when we'll start to see the things 
we never anticipated. Until then we'll just see more middle aged people like me trying to 


recreate our offline experiences online. 
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David Johnson 


VR or AR will allow the combination of text with graphics. The result will be the use of 
inference arrows. Hierarchically nested. VR will allow an infinitely zoomable canvas. And 
the arrows will be computable. And shared spaces will enable debate in the form of iterative 
construction of inference arrow structures. We will see the emergent complexity of our 


collective thought. 
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David G Lebow : Goodbye to My Old School 
Large Format Display 


The advent of VR systems capable of providing the equivalent of a wall screen display with 
interactive features for moving text and images through gesture promises to transform how 
people engage in multisource sensemaking. For anyone who is excited to exercise their 
curiosity by tapping into the internet but feels paralyzed by information overload and 
information anxiety, this is great news. 

Many years ago (i.e., pre-digital age), I remember reading an article with some 
skepticism about how people someday will carry the equivalent of the library of congress in 
their pockets - around 15 million volumes weighing many tons. Today, anyone with a 
connection to the internet and a Webphone in their pocket can access almost everything ever 
published on almost any topic - the storehouse of human knowledge at your fingertips. 
Paradoxically, as the amount of information available to us through digital devices rapidly 
expands, more and more people are accessing this information on tiny screens. 

Back in 1992, as a graduate student in educational systems and technology, I began 
work on my first paper for publication. After reading over a hundred documents in hard copy 
on my topic of interest, I felt overwhelmed by the volume of information. Out of 
desperation, I copied all of the fragments of text that I had highlighted in my 100+ source 
documents to a Word file using my Mac LC with its 12" screen. I printed out the results, 
which included 88 pages of text fragments with document code numbers and page numbers 
for reviewing fragments in their sources. I cut the fragments into snippets, put the snippets in 
a shopping bag, shook the bag to mix everything up, and then used the floor of my office as 
an old school large format display. Sitting on the floor, I reached into the bag, pulled out a 
snippet, and placed it on the floor next to related snippets. As the process unfolded, I 
constructed a narrative-like structure by organizing snippets into columns of related 
fragments and arranging the columns into rows that reflected an emerging storyline. 

After three iterations of organizing snippets, reviewing the results, filling in missing 
information, and looking for non-obvious associations and patterns, I had an Aha moment. I 
saw something that was a new observation. 

Arguably, advances in VR technology combined with large language models and related 
technology promises to deliver on the promise to democratize the creation of new knowledge, 


as first imagined by internet pioneers. Goodbye floor. 
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Deena Larsen : Virtual Reality 


“Imaginary gardens with real toads in them...” Marianne Moore. 


I, too, mistrust it. There are things that are important within our own 
reality. 
Experiencing it, however, from our own imperfect understanding for it, we discover 
that there is in 
it after all, a place for the meaning. 
Hands that can grasp, eyes 
that can see, feet that can move 


and adjust other realities, these dimensions are important not because a 


fascinating new technology can put a glamour on them but because 
they are 
useful; when they become so imaginative as to become 
insightful, the 
experiences may provide visions for all of us—that we 
would not otherwise have come to 
or could not understand. The gloves, 


grasping on in thin air or in quest of something to 


hold, golf clubs swinging, an adventurer storming a moat, a reader harvesting 
these words, 
the program trying new algorithms like muted conversions in dimly lit 
bars, the code— 
(s)talker, the generator—case after case 
could be quoted if 
we build it; nor is it dated 


to cite use-cases for military or education or 
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gaming; all these phenomena are important. One must make 
a judgment 
however: when dragged into commercialization without context, the result 
is not virtual reality, 
nor till the programmers among us can be 
“purveyors of 
real meaningful content” —above 
likes and views and selling and can present 
for inspection, virtual hallways with real people in them, shall we 
have 
it. In the meantime, if you demand on the one hand, in defiance 
of their prophecies— 
the imagination of life in 
all its potential, and 
that which is on the other hand 


vicarious, then you are immersed in virtual reality. 
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Fabien Benetou 


Yet another release around the corner. BigTech is once again promising utopia or at least a 
better, faster, more efficient and creative at the same time future. Chances are it will be a 
partial fantasy and a partial truth. The product will definitely be mindfully crafted by experts 
in their respective field with access to resources most of us barely dare to dream of. Yet, 
behind the promise hides few things we must always be cautious of. 

The first and most fundamental challenge is daring to dream beyond what is offered. 
The product might be slick, might have perfectly finished edges, still it is not finished. One 
could imagine all participants of the project and they would surely admit that they had to deal 
with constraints, at the minimum physics, delivery date and budget, but one can imagine a lot 
more. Consequently whatever is being sold is simply an imperfect version trying to solve 
these constraints. At the moment of its release it is already outdated from a technical 
standpoint. One should thus look at the result of the result as a stepping stone to do better and 
question, not merely accept, what it can actually do and how one could do, with or without it, 
better. 

Secondly and sadly a recurrent challenge is considering the true cost. There is of course 
price as the immediate cost. It is what will already prevent most of us from buying from 
BigTech high-end product that are playing on an elitist appeal. We want to look cool, all of us 
not matter our age or background. We all need deeply acceptance from our peers but even 
from complete stranger. Having, and showing, products that represent the state of the art in 
tech does, from most, just that. They make us look like are already present in the future. At 
least that is the value proposal sold by most advertisement for BigTech. Again this is not a 
complete lie, it does indeed provide radically new usages and thus can help us do more, or 
better. Yet what always comes attached to the price are locks. We know of such explicit locks, 
were for example it is not possible to install an application not part of the store owned by the 
company that made the device. We also know of DRM, or digital right management, that 
make our own devices prevent themselves to act as we may want, think how you can not 
download a book to play on an ebook reader without the right software. What we is more 
subtle though and what we might miss is how these locks interplay. What if instead you could 
indeed bring your ebook or music to another device with ease? Well of course the trick comes 
from the company that made the first device itself and is obvious : buy a second device! Both 
device will perfectly connect together, providing you the smooth experience, something 
almost ethereal where data have no place, where one can "think" across devices. The 


"solution" is apparent and yet we tend to forget of that cost, namely lock-in. The better a 
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device can simultaneously work with "authorized" device from the same seller while 
preventing use from any device, the deeper lock-in gets. The more helplessness we feel as we 


know our dependency on the device and its ecosystem grows. 


Thirdly and finally the most oppressive challenge comes from ourselves, after years of 
consumption, namely learned helplessness. Years after years, using more and more amazing 
products that seemed to complete and that deepen lock-in, we learned to use rather than 
create. Sure we did learn to create within the boundaries of the products we purchased, 
carefully trying to think away from, yet still within a box. We assumed than BigTech products 
are complex, which they are, that they are efficient, which they are, that they are covered by 
patents, also correct, and that making our own would be prohibitively expensive and even 
technically impossible. This though is all partially true and partially wrong. On our own, it is 
indeed nearly inconceivable to watch BigTech release announcement and think "Ah, 
amazing, I could have built that!". This is actually true not just because we do not have the 
resources but because we forget how it is made and its limits. Behind the announcement one 
could imagine the rolling credits of movie, endlessly going on and on, showcasing the 
countless experts involved. So yet, on our own, with little resources, doing a copy of what 
has just been shown with grandiose claims and much glamour is usually unreachable. Yet, 
maybe one part of it, an "easy" part relative to our own expertise is easy. What if we were to 
document ourselves on the state of the art for that part. What if we asked a friend or colleague 
about that other part, chances are for them it might be feasible. Would the final result be 
identical? Surely not but the point here being that just by doing so, we did dare to ask 
ourselves how we could do it, not if. This very step then opens up endless possibilities. The 
product showcased during the release might be amazing but is it truly what fits our needs? 
Surely not as it was done for a theoretical user, not us with our own idiosyncrasies. 

So... yes, releases are exciting but do not forget they are ploys to make one 
intellectually salivate about the next box to put you in. What is more exciting is how WE, 
together, can make transparent documented bricks that connect. We do not need releases, we 


just need to build! 
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Frode Hegland : How VR can, no-will-save us 
from Al. 


In conversation with Barbara Tversky and Anni Rowland-Campbell this week something has 
become very clear: VR (including AR/XR) can provide a powerful environment for how 
to deal with AI, particularly in education. VR environments can provide the space needed 
to unlock the potential of AI to augment how we think and communicate, whereas traditional 
computer displays and speech interactions-though useful in so many situations—severely 
constrain our interaction and therefore our involvement, and ultimately, our control of AI. 
The following is a review of our possible near-futures and a suggestion and plea that we take 


it seriously and invest in augmenting how we think and communicate. 


deep & shallower 


From a perspective of valuing ‘Intellect Augmentation’ (IA), in the Doug Engelbart 
tradition, as any new and powerful thinking and communications technology, from paper to 
print to the Internet, AI can both increase the depth of our thinking and also make it 
shallower. It will all come down to how and when students, academics, and people in general 
use it. Engelbart presented the notion of augmenting human intellect in his seminal 1962 


paper, which would be the basis for his ‘mother of all demos’ in 1968: 


“We need to improve how we augment a group 5 (small, large, internal, global etc.) 
capability to approach urgent, complex problems to gain more rapid and better 
comprehension (which can be defined as more thorough and more critical) which result in 
speedier and better solutions (more contextual, longer lasting, cheaper, more equitable etc.). 


And furthermore, we must improve our improvement process (as individuals and groups). ” 


Doug Engelbart [1] 


fast food 


Anni made me consider what many people have been doing lately, thinking about AI’s impact 


on human thought, particularly as related to education. 


In a sense, AI can be seen as being mostly in competition with IA in much the same 


way heavily processed, fast-digesting sugared fast food is in competition with healthier, more 
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whole food which the body needs to take time to digest; more fibre, more bran. Many of us 

will mostly prefer the fast food. And in many cases this makes sense, there is much thinking 
we do on a daily basis which is not really helpful, such as remembering phone numbers and 
turning on hand brakes (electric cars do not need them) and so on, tasks which systems have 


already removed from our daily grind. 


Historically, sugar is not all bad. After 1800 imports to Britain soared and the change 
from a fibre-heavy to a sugar-heavy diet, noted Robert Fogel, a Nobel prize-winning 
economist, “raised the proportion of ingested energy that [could] be metabolised”. 
Furthermore, Fogel reckoned that “bringing the ultra-poor into the labour force [and] 
raising the energy available for work by those in the labour force” accounted for about a 
third of Britain’s economic growth in the 19th and 20th centuries. Will AI augment more 
people to be more productive? For much work, it seems that AI can at least remove some of 
what slows us down, the daily grind, maybe clearing up enough mental load to give us more 
capacity for further mental exertions. 

For what we consider important information we’ll need more ‘brain-bran’. More brain 
involvement for the brain to truly understand & be involved. The breakfast cereal analogy is 
not entirely silly, the idea is making information great for digestion and integration into our 


current knowledge, as well as palatable (pleasant to interact with/providing ease of use). 


outsourced thinking 


When it comes to augmenting our intellect, a lot more will be outsourced to AI in the future 
than it is now-or perhaps the term should be ‘delegated’-that much seems inevitable. Much 
of this outsourcing will be of value, similar to how we could search the Web for answers 
before, we didn’t need to go down to the local library. There will also be issues of actual 
dumbing down where people will not want to learn but get the information in a presentable 
form to solve a problem, such as that of a school paper, keeping the student’s knowledge 
shallow. We might also learn to outsource in ways inspired by the octopus’ tentacles which 
operate at a level of independence so far alien to us [2]. 

On the other side of the potential of AI, to increase out depth of knowledge, I heard a story of 
an old grandfather hearing about the recent advancements of AI and his response was 
something along the lines of “this means everyone will have access to great intelligence, it 
will almost be like we will all have a genius best friend” and that is nice. And I heard this on 


TikTok, another disruptive platform and interaction. 
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linearity & non-linearity 


Human thought is multidimensional, as knowledge itself is. It can be argued that academic 
discourse is based on linear arguments however. A student cannot simply drop a knowledge 
graph or a pile of index cards on the desk and call it an academic paper. Well—written 
academic papers, at any level of education, are intended to make clear the author’s thinking, 
something which AI has become adept at mimicking. Augmenting the documents to enable 
the reader to unlock the non-linearity of the author’s thinking has been a key aspect of my 
work on Visual-Meta and now this approach looks like it can help with the problem of 


knowing if the student truly wrote a paper. 

If the teacher, advisor or even journal editor had access to the author’s connected 
knowledge network, connected with the paper, they could use it to check the reasoning and 
connections. Not by itself, but as a connective tissue of the main work presented. 

At this point in time we need to ask what we want our educational system to do. 
Education has always been a reflection of what those with the means to control it would like 
it to be. 


teaching is what society wants taught 


How we feel about the current state of education will very much come down to how we feel 
education fits within our political views, where the Left seem to have more affinity to higher 
education and the Right being more suspicious. This is nothing new, education has always 
reflected what society wants. 

During the 1400s, 1500s and 1600s when Europe’s universities were growing, teachers 
were growing in demand. An illustrative example of this is Bologna, in Italy during the 1500s 
where there was a high demand for notaries, officials and secretaries to cater to the 
institutions of the elites as written by Andrew Pettegree in The Book in the Renaissance [3]. A 
result of this the profession of teacher increased to fulfil this need, where what was taught 
was what was needed to perform this work. In France invention of print started to challenge 
the preeminence of cathedral schools, where many municipalities appointed their own school 
masters in response to church control. Although starting with basic literacy, by the from the 
1500s these schools also provided a more liberal arts education and the institutions grew. In 
In Bordeaux it was proudly proclaimed that the new ‘collége’ would “increase the profit and 
honour of the city”. Pettegree further highlights how citizens were proud of being able to 
educate the young in their own town, though this should be done in the ‘Parisian Style’, 


helping bring uniformity to the teaching profession. ‘ 
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whole human thinking 


Back to today, we—as a society—have to choose to enable our educational institutions teach AI 
as a component of IA, and support how they can do this. One way is to make the linear to 
non-linear interactions more tangibly real, we need to keep cultivating the inclusion of the 
whole human in human thought, not focus on only the brain accessing information through 


small rectangular computer ‘screens’. 


Barbara helped me consider the layout of information. I introduced her to our new, 
streamlined ‘Library’ in Author and Reader and she pointed out that when she works she has 
different categories of information in tabs in a browser, in various open documents and so on, 
which helps her have a sense of what’s where. In The Extended Mind [4], Annie Murphy Paul 


presents numerous examples and perspectives on this. 


This is so very important. We should not simply outsource thinking from our brains to 
AI, we should move thinking out of the exclusive domain of the perspective that it only 
happens in the brain and accept and embrace how our full our bodies and environment is the 


space of thought. 


transformed text 


Although AI already produces powerfully compelling speech and imagery, I believe there is 
still space—and need for—text. Text has been uniquely transformative for human thought for 
millennia, and can continue to do so, if we work to truly unleash text, including AI powered 
text, in richly visual environments. I do not fear a ‘textpocalypse’® but uniform columns of 
black on white text blurring into grey has probably had its day, outside of very specific uses, 
text will have to evolve to stay relevant in our work and daily lives. 

Text will have a profound impact on our future with AI and VR, if we work to make it 
so. If we do not, text will drown itself through producing too much of low value and 


producing little of use. 


But why text, in a world of powerful AI where we carry on vocal interactions with our 
AI systems and receive information in video form? Please forgive this perhaps a little 
defensive section on why text matters. Feel free to skip it if this is obvious to you already. 

Text VS. Voice and Video. A unique nature of text is its fixedness and instant sequence 
overview in a way which other media, such as voice and video cannot provide. 

The expectation of coherence in a written work is high where as the exception of a 
discussion is quite the opposite, the expectation is a live probing for understanding where the 
participants speak what is necessary to be understood, with the expectation of the listeners to 


ask questions to build a more complete mental model of what is said. A written piece has the 
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expectation of being edited to some degree, or at least having been ‘looked over’. A recorded 
video is a series of images with little help for editing, which is why when editing video with 
speech, new systems which allow human speech to be re-created, uses text as the interface; 
“It takes only a few minutes and you'll be turning any text into natural-sounding Voice Over 
audio ”4. 

When we prepare a spoken presentation for anything longer than what we would say in 
a back-and-forth conversation, we still refer to having a ‘script’, we still use text to note 
down our thoughts to make sure we present them in a ‘thoughtful’ order. I can testify to the 
mental effort of producing a coherent linear text, such as the one I am working on for you 
now, and my PhD, still in its Corrections phase, as being mentally taxing, requiring copious 
amounts of water, coffee and maybe some chocolate and protein bars to complete. Writing 
anything longer than a sentence is hard, but it is considerably more achievable than not doing 
so, by trying to do it all ‘in my head’. 

Text therefore offers different dimensions of knowledge traversal, able to connect in 
multiple dimensions. Emoticons will only ever achieve a different richness than what text 
provides; more amusing, more visual for a short ‘sentence’ but less parseable in longer form. 

Still images, moving images—both captured and synthesised through 3D or Al-are also 
powerful communicators in skilled hands and will become even more so over time. There is 
also nothing to replace a face to face discussion. I am not writing this to put down other 
aspects of human communication, though I do ask you to consider how this document would 
appear if spoken, as sound only or with images. It would be handy to listen to as a podcast 
episode but it would use a single channel to your brain, the audio stream or sequence of 
images. As text you need to make an effort to read it but you gain instant access to the 
overview of the text, your eye can jump around on the display to see connections and jump 
back to what you might now understand in a different way and more. Just as I would love to 
discuss the issues here over coffee or over Zoom, the record of such an interaction would be 
rich in human interaction but poor for access for those who would later listen in, as my 
experience in 200 sessions of the Future Text Lab can testify. 

Speech matters powerfully but it serves a different need and leaves a different record to 
what is written. 

Text VS. Images. What text uniquely bring to the table compared to other visual 
images is how text is a series of ‘grammatical symbols’ which are fixed for the user, not ever 
flowing like video or speech. Reading allows the eye to jump around and reflect on different 
parts of the text, pause at will and not have lost any context. 

Text, though visual, transcends the specifically visual. The word ‘squirrel’ captures all 


types of squirrels in all types of situations and concepts. A drawing or a photograph of one, is 
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just that, one specific squirrel in one specific setting. It is often reflected upon that ‘the movie 
was not as good as the book’ and this is because the book was projected in the readers 
imagination, with just enough detail and vividness to create an immersive experience, 
whereas a movie ‘pre-rendered’ everything, never allowing for a a scene to happen without a 


previously arbitrary background not to be filled in. 


Digital Text. There was a powerful transformation when text moved from being 
primarily analog to digital. When we left working on paper for working on screens, we 
gained interaction in depth but lost interactions in space. 

When working on paper we could easily move several pieces of paper on a desk or 
stick them on a wall, even stretching thread between them to show relationships as so many 
TV shows illustrate. The interactions digital systems affords us provide rapid access to 
resources, all within the confines of the space of a regular size book, opened on our table. We 
can no longer dog-ear a book and look at it on the side to see what sections are of interest. 

Writing and image making has become separate, with ‘word processors’ and ‘drawing 
programs’ occupying separate places on our screens and using different file formats®. Text, 
images, video and audio are not more powerful than the other, but become more powerful 
when usefully combined. 

The unique nature of digital text is in its ability to instantly change what is displayed- 
flexible fixedness. 

The cost of a digital display—though continually falling—is significantly higher than that 
of paper and is practically restricted in ways paper is not. A visual artist used to carry around 
a large portfolio case to show their work, now it’s on an iPad or on the Web, along with the 
clutter of the Web Browser interface and the client’s other tabs and windows. Now a portfolio 
can provide instant access to innumerous images and also include video, but unless the 
viewer uses a large display, the image will be as small as a printed picture in a frame on the 
table. 


Text not only lost space, but also texture when it became digital. Text on paper can be 
whatever size or shape you want it to be, and it can overlap and part of other media in 
anyway you prefer. Pen on paper, in the hands of someone with the required skill, offers great 
freedom of expression. 

This all brings to mind the modes of hypertext and discussions around tools for 
thought: text for thinking and text for communicating are very different things, with an 
overlap that can get messy. Text for presentation may very well in many or even most cases 
be more useful if rendered into speech or video whereas the ‘authoring’ or writing still stays 
as text. Conversely, reading the text may very well be best as audio or video, though recourse 


to a written form may be better for examination of the text rather than the time based audio 
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and video where the text, in a sense, ‘washes over us’. 


dimensions of thought 


Just as there was was a powerful transformation when text moved from being primarily 
analog to digital, there can be#if we work to realise it—another powerful transformation when 
text moves from two dimensional substrates to full-space when experienced in a headset. 

VR can provide a rich experience for the next stage of interactive information spaces. 


VR has none of the constraints of portable little rectangles, large and fixed rectangles or 


physical substrates. 


When the quality of the headset display becomes good enough', we can have the best of 
both worlds: Display surfaces in the virtual world we can scale to any size, put in any 
location and connected in any way we want, at low effort and zero additional cost. Not only 
will we be able to expand on Barbara’s workspace, we will be able to transcend it, shaping 
our knowledge into truly useful forms. We have long been able to take inspiration from 
science fiction [5] [6], and now, even in mid-2023, we can start building and experimenting 
to experience. I started working with digital video when 320*240 pixels was considered 
pretty good and 640*480, which came soon after, was considered standard video. My current 
video camera is capable of 4K (4096*2160 pixels) and I’m eagerly eyeing 8K (7680*4320 
pixels) which is no longer in the realm of only major production houses. I expect the same 
will happen to VR and AR experiences. The current Meta Quest’s 1440*1600 will soon be 
overtaken by Apple’s Reality Headset’s 4K (4096*2160 pixels). This is to say nothing of the 
computing power which drives the displays. Currently Apple’s Watch is capable of 
immediate translation with stunning accuracy and speed. 

There really is no question that VR will appear very, very real, very, very soon. The 
question is simply whether we will let this opportunity to truly invent new ways of thinking 
to get the support it deserves, or whether we will revert to re-creating the physical world 
digitally. 

To be clear, I do not see AI as only a threat, a dumbing-down system, a propaganda 
machine or a way to produce what some has called more grey goo text®, though all of those 
scenarios will likely play out to some extent. AI can also greatly help us take advantage of the 
additional dimensions VR will afford us. AI will be able to support these views, arranging 
what is shown and how it is shown with greater ease than ever before. 

To use some visual analogies AI will not be out of the picture if we make the canvas 
large enough. AI will box-in our thinking, if we frame it too small, if we interact with it 


through too small an interface. 
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What will be new will be the ways the author can thread together an argument and 
present it in a way the reader can unravel and view in multiple dimensions, making it more 


intuitive to judge and experience the information. 


We need to pursue augmenting human thought with artificial intelligence in a virtual 


environment for a full brain and bodied, a fully human human experience. 


This will not only help free us from the potential of AI thinking for us, not with us, but 
it will also allow AI to augment us more powerfully. 

To think is to move [7]. Thought is not static. Will we take this historical opportunity to 
employ AI in VR to super-charge how we think and communicate or will AI take over 
thinking and VR become mere entertainment? Will we be able to unleash deeply ancient 
aspects of what it is is to be an embodied human in a completely modern, artificial 


environment, to allow us to navigate this ‘cyberspace’ with all our faculties employed? 


‘combined arms’ 


Several years ago I wrote how we should ‘arm’ the citizenry in combat misinformation, 
propaganda on one hand and increase understanding and communication on the other hand. 
Here I propose that we accept the issues we face and employ all the media available to us to 
truly augment everyone. 

The only form of control which can allow us to thrive in the face of the issues facing us, 
including both those of the world and those of the media we use to communicate with each 
other about the world, is truly distributed control. Imposed and directed control will over time 


destabilise and change actions into reactions. 


Augmenting text to augment how we think and communicate, with AI in VR, can 
profoundly lift our species. 

There really is also no question that VR will completely replace traditional digital text 
any more than digital text completely replaced analog text. Sometimes a tablet or a laptop 
will be exactly what you need and the headset will feel like an encumbrance rather than a 
liberation. Sometimes a paper book will present exactly the right reading modality. At other 
times a glance at your digital watch will be all you need—the smallest screen you own will be 
enough. To truly thrive in this new world of opportunities it will matter how we can take our 
information with us. Text in VR and text on your watch will have profoundly different 
affordances—there will be very different ways you can read and interact with the text—but it 
should retain its richness when you choose to access it through one medium or another. A 
work session in a VR room with your full library available and all your walls covered with 


relevant information which you then synthesise into a coherent and concise ‘document’ for 
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others, should be accessible anywhere for those who will read the text. 


The Future of Text 


The approach we are taking to make this possible is for a document to have what it is-who 
authored it, what it’s structure is and how it connects—spelled out in an appendix at the back 
of the document. We call this Visual-Meta and even though it can maintain all the 
information through a document lifecycle which can include version updates and even 
conversions to other formats, and even printing to paper and scanning, it can also retain 
information in and out of VR environments and provide AI with ‘authoritative’ data through 
which to work from. Cyberspace, or the ‘metaverse’ should not exist isolated from the rest of 
our information environments, it should not exist in a ‘bubble’, but stay richly connected. My 
professor and dear friend, Ed Leahy, to whom this article is dedicated, passed away last week. 
He was that rare teacher who did not instruct and tell us what to do, he continually questioned 
and helped us think things through. Over the last few decades of focusing on the future of 
text, where my mentor Doug Engelbart encouraged me to pursue looking at what he called 
‘symbol manipulation’, I have edited and published three compendium volumes of over a 
million characters, hosted the Future of Text Symposium for over a decade, many co-hosted 
with the legend that is Vint Cerf, it is clear that the future of text will be the result of 
questioning every aspect of what it can be. Questioning of what text should do for us and 
how text can be employed to most powerful effect in all its forms. 

Anything less will see text both overwhelm us and be fully displaced by more easily 


persuasive media. 


The future of text has not been written. 
Barbara Tversky Response 


We have to think of thinking as an interaction of the mind/brain, the body, and what we sense 
and create in the world. Thinking is certainly not limited to text or to words. 

Remember sensory deprivation? People with no sensory input hallucinate (they're in 
immersion tanks with eyes blindfolded and ear plugs). 

AR/VR can do some of the world part, though engineering what should be in the AR/ 
VR is far from trivial and people often don’t have insight into what would help them think 
much less what would help others. AI might be able to help on that, depending on how the AI 
is trained and updated. AR/VR currently lacks good haptic feedback and in many cases 


doesn’t give a 1-1 distance experience. When people navigate the real world, they have a 
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sense of how much time it should take, when the destination or intersection should be 
reached. 


Place, where things are in the world, is part of their meaning. We “read” graphs from 
left to right, we understand them faster and more accurately when “more” or “higher” goes 
upwards. A stable world is also important; we remember where we put things, books, 
clothing, kitchen utensils etc. are without looking and rearranging takes time to learn. For 
years I would catch myself driving to my old house at Stanford instead of my current one. 

I am puzzled by “save us from AI.” Do we need to be saved from AI? It certainly can 
be helpful as well as harmful depending, but I don’t see how VR can save us from the 
harmful potential of AI. 


PS. 

It’s not text vs, image; if anything it’s integrating them, comics. See Scott McCloud’s 
Understanding Comics, see Larry Gonick’s wonderful cartoon guides to statistics, genetics, 
history, calculus, chemistry, physics, and more, see hundreds of graphic novels that work by 
integrating all kinds of text with all kinds of images. 

The world communicates; it’s a diagram in space (see the last chapter of my book): 
lines separating lanes for cars, buses, bikes, parking places, cross walks; arrows, pictograms 
of all sorts on signs and the roads; stop lights; the other things moving on the roads and 
sidewalks; sirens. All that guides users of roads and sidewalks rapidly and almost 
unconsciously, 

Personally, I think AR is going to be more useful in navigation and in tasks like 
assembly, cooking, knitting, preparing documents, sporting events, museums, sight-seeing 
especially for historical additions, and more before immersive VR takes off but Andreea (and 
perhaps Anni) has more experience with what VR from a chair can do. 


So much to think about and talk about! 
Frode Alexander Hegland Response 


Absolutely, fully agree. 
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Frode Alexander Hegland : Walkthrough of a 


Dream 


As I sit here in my office upstairs, at the back of the house in Wimbledon a very early 
Tuesday morning, working on my Apple Laptop, a 13” M1 Pro machine, using Apple’s 27” 
Studio Display, my iPhone 13 Pro on the desk, windows open to the bird and fresh air coming 
in through the window, sipping both coffee and water, Edgar sleeping next door, Emily 
waking on the other side of the house, I can imagine putting on my headset and continue to 
work. 

I am an unashamedly Apple fanboy who read MacUser when I went to sleep when I 
was younger. I count Keith Martin, MacUser’s technical editor as a close friend, same as 
Bruce Horn, author of the original Mac Finder. I am lucky enough to have hour long 
discussions with Brandel Zackernuk ever week. But I am still very concerned at how Apple 
will work to own and turn the headset into a fashion item for entertainment, sport and 
socialising. But never mind, I am writing about a dream, not a nightmare. 

I put on my headset and continue to work. 

What I was working on-the text-I am writing to you now, in my Author software—on 
my Studio Display, is still right there in front of me, only now it’s projected onto the view, 
perfectly aligned with the physical display. Since the room has been thoroughly scanned with 
the high resolution LiDAR Scanner in my iPhone, I can choose to not only make this virtual 
display larger, but also smaller, it can remove the physical display. 

I continue writing this document and the room I was in is much the same as in reality, 
apart from a few ambient systems giving me information about what’s going on in the world. 

I make my display a little smaller, a little taller, exactly what I want for writing, just 
using my hands and pulling on the virtual frames. I then pull out the Map of the document 
and place it on my map wall, as I have dedicated it. I love how it is connected to my full 
knowledge Map, which is available through a few gestures. 

This then illustrates my first wish, to be able to have as many traditional windows 
as I like, and place them where I like, including onto named places, such as a Map wall or 
Photo wall. 

I then have a meeting with Fabien and we appear as talking heads for most of the 
time, so that we do not have to be self conscious about small movements like reaching across 
for a coffee and so on. When necessary we can enter the space fully, to present something on 


a virtual screen for example. Fabien can see all of my room, as I haven given him permission, 
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it’s like he is in my physical office. When we work we share documents as though they are 
physical documents, and just as easily. He asks me to have a look at the VR environment he 
is building and I join him, as easily as walking through a magic portal. 

This sounds pretty simple but there are a lot of aspects needed to make this work. And 
we should make it work. How to share data, how to enter and leave spaces and to what 
degree we choose to be present. 

In the further future I don’t know what I want. But I need to experiment to experience 
to understand. And this can only happen when we have an actively open environment in 


which we develop. 


The Politician and the Headset 


I show the headset to a politician who ‘doesn’t get this VR thing’ and when it’s on her head 
she is immediately struck by how immersive it is. I then present her with her laptop’s screen 
and she experiences what I showed you above; I change the screen, move things into the 
world and have a meeting. She takes her headset off and says this is not what she expected, 
she expected a more of a toy world with Zuckerberg cartoon people. I tell her about the need 
for open development infrastructures and she says the market should decide, this is too hard 
to regulate. I then tell her what she saw was all completely fake for a demo and she takes the 
headset back on and tries the real systems and after a while it becomes clear that the lock in is 
worse than it was with Microsoft Office and sharing information is nearly useless. This is 
where I see politician will start to flicker in her eyes. 

Of course, I can’t afford to develop such a system to make such a presentation, but can 


we afford the opportunity cost of not pushing for more open development worlds? 
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Leon van Kammen 


The next step in VR technology, in my opinion, is for it to become hyper-navigable, 
addressable, and shareable. 

This is not a feature that app stores can provide, because they are designed to focus on 
separated, mono-curated, mono-layered experiences. 

If VR operating systems are to be appealing, they need to offer the opposite as well. 

One low-hanging fruit to achieve this is for VR headset manufacturers to enable certain 
multi-layered experiences by default. 

This would allow the app store layer and the web layer to coexist, enabling users to 
access notes on the web while experiencing an app store. 

Even by restricting the web-layer to plain HTML without scripts & rich-media would 
unlock enormous multi-layered potential. 

By doing so, VR/XR device manufacturers can take advantage of innovation waves in 
multiple directions simultaneously, instead of ‘waiting for the next killer app’. 

Additionally, a single solo-button that hides yourself/others should be added as standard 
privacy-equipment, making multi-party experiences always optional aspects of any VR 
experience. 

As most people read books or write emails in solitude, this would open up greater 
opportunities for introspection, cognition, and meditation. 

To complete the circle of text unfolding itself to (VR) experiences and vice versa, we 


need layers that promote these conversions. 
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Mark Bernstein : Writing On The Wall, Text In 
Virtual Reality 


a contribution in honor of the fifty-fifth birthday of Frode Hegland 


The writing is on the wall: virtual and/or augmented reality is real. Or, perhaps, it will 


become real. What is the future of text in the virtual world? 


Fiction anticipates virtual screens will be much like physical screens, or the projected data 
surfaces of Starfire [8]. Here’s Arkady Martine’s vision of a day at the office in the heart of 


the galactic empire: 


The interface was....more intuitive than Mahit had expected. She gestured and 
the Infograph responded—spreading her hands and twisting her wrist spawned 
multiple transparent workscreens, and she could make her own halo of 
information. She found Nineteen Adze 5 preset camera feeds and called up the 
one that was still trained on the demonstration....and set it to run off to her right 
side. Over her left should she put a window full of a running stream of tabloid 
headlines, and resolved to improve her vocabulary of casual and insulting 
vernacular—and perhaps also to learn something more about anti-imperial 
activists. 


Martine Arkady [10] 


Martine’s Empire has planetary cities, cheap interstellar travel, a history spanning thousands 
of years, a culture and politics of Byzantine sophistications, yet their screens look a lot like 
our screens. Indeed, beyond their translucence, they’re not very unlike Murray Leinster’s 
“Logics” of 1946. 


It looks like a vision receiver used to, only it's got keys instead of dials and you 
punch the keys for what you wanna get. It's hooked in to the tank, which has the 
Carson Circuit all fixed up with relays. Say you punch "Station SNAFU" on your 
logic. Relays in the tank take over an' whatever vision-program SNAFU is 
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telecastin' comes on your logic's screen. Or you punch "Sally Hancock's Phone" 
an' the screen blinks an' sputters an' you're hooked up with the logic in her house 
an' if somebody answers you got a vision-phone connection. But besides that, if 
you punch for the weather forecast or who won today's race at Hialeah or who 
was mistress of the White House durin' Garfield's administration or what is PDQ 


and R sellin' for today, that comes on the screen too. 


Murray Leinster |12] 


This stability of (partially-occluded) vision might reflect our impoverished imagination, but I 


suggest it points to some facets of text that we seldom have occasion to discuss. 


Visual Field 


I have argued for nearly 25 years now that our urgent goal must be to invent forms of writing 
that are “better than books” (Bernstein, 1999). The codex we know does not meet our needs, 
and never has; it is neither divinely ordained nor the end-product of two millennia of 
development. It finds itself , as it always does, at the rapidly-changing intersection of 
numerous economic, intellectual, and social force (Bernstein, 2023b) (Bernstein, 2023a). 

Yet in one respect the contemporary codex comes close to being everything it can be: 
books occupy just about as much of our visual field as we can spare. We know precisely how 
small we can make our letters, because we are deeply interested in some codices that were 
originally conceived not something to be carried in a pocket but as a library to be stored in a 
dedicated room, maintained by a dedicated community (Amory, 1986). If we could make the 
text in the Oxford English Dictionary smaller, we would (Raymond and Tompa, 1988). A 
quarto block of 30# book paper weighs about 0.75 1b/100 pages, and if this book was part of 
the documentation you need to keep on hand aboard a commercial passenger plane, the 
aircraft designer had to anticipate the fuel cost of keeping those pages almost continually 
aloft over a span of 30-50 years (Malcolm et al., 1991). We’ve known since before the 
beginning that we’d like print to be smaller and the page to be bigger, but we simple cannot 
manage that (Buckland, 2006; Bush, 1945). 

The glowing informatic halos of interstellar empire resemble the screens we know 


because they must. 
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A Library Table Of One’s Own 


When I was an undergraduate at a small American liberal arts college, some students each 
appropriated a personal table in the school’s small but under-utilized science library. There, 
we'd array books and notebooks, textbooks and assignments, sources lying open to await our 
return. This was great, but you couldn’t get away with that at Harvard, where even the most 
obscure libraries were far too busy. 

Those library tables, though, might indicate something that virtual reality could give us: 
ready access to assemblages of information we have laid out for use, residing at the edge of 
our vision or conveniently beyond it. Indeed, the familiar map views of VIKI (Marshall and 
Shipman, 1997) and Tinderbox (Bernstein, 2011) seem at least a gesture in this direction, yet 
they are far less informative (and far less comforting) than a table full of references set out 
before us. The book wheel is an oft-cited but seldom-used anticipation of intertextuality; one 
survives in the Strahov Library in Prague. Virtual space might well accommodate 
comprehensible assemblages, laid out for our convenience, to which we might occasionally 
admit a colleague or collaborator. 

Reading multiply is a useful skill: histories from Gibbon (see (Grafton, 1997)) to 
Lekson (Lekson, 2018) may best be approached by reading two copies at once: one open to 


the text and one to the footnotes. 


Written Above The Door 


The sign above Hell’s Gate instructs the viewer to abandon hope. Orpheus ignored it 
(Mitchell, 2020). So did Dante. Quite a lot of what we write on buildings is not meant to be 
read, or at any rate, not meant to be understood. 

Even when we write on walls with the intent to be seen, as in graffiti, it is far from clear 
what, precisely, we wish to express. It mattered to Don Diego de Vargas that he had visited 
El Morro in what he had decided to call New Mexico in 1692. But what reader is envisioned 
here, and what is that reader expected to conclude? 

“The moving finger writes”, Edward Fitzgerald reminded us in Stanza 71 of his 
Rubatyat: the writing on the wall can matter. But from Lascaux’s horses to Mucha’s art 
nouveau actresses, public inscription is often ambiguous, ambivalent, and wayward. The 
other epigraphic tradition, of course, involves things written on stone and displayed in public 
as an deterrent to corruption and special-dealing: anyone can read (say) the Athenian Tribute 


Lists, the Fasti of Rome, or the Laws of Hammurabi (or Moses), so their content is widely 
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known. Additions and forgeries are difficult because they’re written in stone in the middle of 
a busy marketplace. 

This matters to virtual spaces in two ways. First: stability. If writing persists, it marks a 
place not only for us but for everyone else. Conversely, if we know writing will not persist— 
if it will be burnt at the end of the festival or washed away by the tide (like tears in the rain) 


— that instability can inspire memory or reinscription. 


In the winter of 1983, I stood on a barren hillside hundreds of feet above the coast of 
the Black Sea, 

and I promised myself that yes, I would take care to record in my mind, and later 
remember, 

the time that was to come, each and every moment of it. It was a promise I kept well. 


Magdalena Donea, Moments 


Secondly, there does remains an architectural surface on which people inscribe ephemera: the 
chalkboard. The utility of blackboards for instruction has largely been superseded, but in 
mathematics they remain an important resource for working through tentative, inchoate ideas. 
Chalkboards work because they are easily erased, and because their scale is suitable for 
collaboration (Wynne, 2021). It has long been recognized that virtual spaces lend themselves 
to rescaling, and that this could be valuable when working with large sets of equations or 
large sets of data that cannot adequately be represented as a statistical ensemble (Asimov, 
1953). 

Yet, once more, we are left with a virtual representation of the familiar: a plane (or, at 


any rate, a continuously-differentiable surface) subject to simple conformal mapping. 


Architectonic and Gin 


When Bolter described hypertext, he used the term “architectonic writing,” which is to say, 
writing with (or on) structure (Bolter, 1991). The choice of adjective has not inspired much 
comment, though (Kaplan and Moulthrop, 1994) holds up rather well. Bolter’s 1992 
Hypertext keynote discussed hypertextuality in terms of museums and memory palaces, an 
approach that seems well suited to virtual spaces (Hargood et al., 2018). Yet physical spaces 
present significant obstacles and difficulties; the audience can see things too soon, or miss 
things because they’re on the wrong side of the hill or too far from the parking lot (Millard et 
al., 2013). 
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Why architectonic and not architectural? One reason, of course, is that the term 
“architect” was the focus of a prolonged struggle to establish a profession analogous to 
physicians and attorneys. Anyone who builds can be a builder, a tekton, but only a 
professional can be admitted to the rank of architect—and only at the call of established 
architects. The point of “information architecture” (Rosenfeld and Morville, 1998) was that 
professional information architects would run things and would rein in the exuberance of 


waywardly unprofitable creatives. The less-familiar term “architectonic” sidesteps this issue. 


More interestingly, “architectural” has acquired a vernacular connotation of decoration 
or ornament, as in “architectural millwork” or “architectural lighting”. Architectural writing 


might be writing on the wall. 


From time to time, hypertextualists remind us of Sullivan’s dictate: form ever follows 
function (Sullivan, 1896). We ought to remember here what Sullivan built, at least as much 
as what he wrote. The bank of Owattona, Minnesota did not insist on those chandeliers: 


Sullivan did. It might repay us to contemplate why they are as they are. 


Structure 


Our hypertext maps are, proverbially, boxes and arrows, expressions of structure that strive 
for legibility, and in which the the complexly tangled web view of Intermedia serves as a 
terrible warning (Conklin, 1987). 

All scientists believe that the observed world is structured—that it is not arbitrary or 
random, that is is (at least sometimes) knowable. The hypertextual tradition grows, in part, 
from the early 20th-century effort to formalize knowledge, to set mathematics and philosophy 
on a sound foundation (Budiansky, 2021). That effort revealed that, in fact, not everything 
that is true can be proven, and in doing so established clear limits on what would become 


computer science. 


Everything is intertwingled, and always has been (Nelson, 1974). Not all hypertext 
maps are neat. Our representation of the world ought to be as simple as it can be, but not 
simpler. 

Our visualizations valorize clarity, but clarity is not always to be had. Obscurity might 
indeed be our own failure, but sometimes things are obscure because the moon is down and 
the Pleiades set. If we find ourselves (as we do) on a darkling plain swept with confused 
alarms of struggle and flight, that’s the way it is. It is neither our fault nor the punishment of 
our sin. It is the way the world is. Some things cannot be measured (Heisenberg) and some 


true statements may be unprovable (Gödel); that came as a distressing shock, but we’ve had a 
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century now to recover. The world we see — say, in Sargent’s Daughters of Edward Darley 
Boit — is replete with dimness and darkness, lost edges, with faces turned away from us or 
hidden from sight. Many of the demarcations and boundaries in our idea planes might not be 
needed once we accept that perfect clarity cannot be attained. Removing them might well let 
us see better. Non-Euclidean geometries warp and transform the plane unexpected ways; in 
particular, the hyperbolic space of the Poincaré disk (and the Gaudi roofline) neatly 
accommodates the combinatoric explosion to which taxonomies and tree structures are so 
often inclined. (Bernstein, 2023c) 
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Patrick Lichty : Methods of Al & VR — Stitching the 


Bell Curve 


Al is changing the fabric of reality if one considers the totality of amassed texts within 
mediated environments as the fabric of encoded experience. One of the main concerns of the 
coming of AI is outsourcing cognition to synthetic agents. This reflects the shift from the 
spoken to the written story through the allegory of the Egyptian god Theuth and the Greek 
king Thamus.[2012] In return for written language, the price would be that of oral culture or 
the remembered record. However, historical accounts tell of relatively factual information 
retained for generations through oral culture. Even with the transcription of scribes in the 
Middle Ages, there were copious notes in the margins and alliterations between 
transcriptions. The abovementioned issues bring forth two issues, translation, and veracity, 
before even going to notions that Hegland put forth using VR as a cognitive filter for artificial 


intelligence. (2023) 


The first translation issue has different elements yet to be addressed as of June 2023 
(the time of this writing) regarding artificial intelligence engines using LLMs, such as GPT4. 
These errors relate to the relationship between the large language model and the translated 
input accessing it. This relationship, while given base parameters by the program, is defined 
mainly through the training and self-programming of the engine. This process could be 
better, and semantic associations with the translator may not make sense to the human 
operator and are mainly a result of the internal consistencies of the model. 

Secondly, veracity is one of the main issues with machine learning and LLMs. This 
problem is relatively simple and is explicated easily by a query for biographical or 
bibliographic information, with the engine making up several cases, books, or non-extant 
articles. (2023) This is analogous to trusting oneself to a calculator, rather than doing the 
manual calculation, and having the answer for "3+5" equals 7.325, and the human operator 
saying, "Well, that's close enough...." In this writer's assessment, the willingness to accept 
less than accurate representation by technology opens humanity up for nothing less than the 
inaccuracies in shipping logarithms that resulted in imagining Babbage's Difference Engine 
in the first place. (2002) With such an acceptance of unreliable narration for the sake of 
imagination, the process is potentially disastrous. Even if it attributes itself to the complexity 
of the engine, when a lack of veracity is proven, it should not be relied upon. 


The flaws inherent within contemporary artificial intelligence bring us to an initial 
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stopping point for the widespread adoption of machine learning and large language models 
concerning their reliability. Microsoft's adoption of artificial intelligence in the Bing engine 
(2023) and elsewhere has already ignored these concerns. Again, this writer contends that 
this is a function of capital's need for early adoption versus the lagging time frames of 
legislation in the nation-state, which is only partially contiguous to infopower. Once this is 
said, the superstructure will likely adopt machine learning before it attains a suitable level of 
veracity/reliability. 

The next issue is, as Hegland states, the inverse bell curve between Artificial 
Intelligence (AI) and Intelligence Augmentation (IA), referring to Engelbart. Engelbart’s 
intelligence augmentation model called for improving computer systems, which would 
challenge the human organism to improve itself to work more effectively within it, creating 
the next evolution in computation. (2001) Artificial intelligence merely takes on more 


cognitive load than "user-friendliness" and other human factors. 


One solution is to change the regime of human factors when managing computation. 
One idea is that of using virtual reality, but having been a user of Microsoft’s Hololens, two 
interface regimes come to mind: spatial computing and Augmented Reality or XR. 

The interface regime of spatial computing, as shown through the Hololens, is scanning 
the local environment and using that as where one can place holograms, virtual desktops, and 
interface components. A critical element to this is being integrated into XR headsets, such as 
the Meta Quest and Apple glasses, which is hand gesture recognition. If a computing regime 
evolves from the keyboard, mouse, and screen to that of virtual representations of 
computational entities, browsers, holograms, or otherwise, a natural interface regime would 
include that of manual control using one's fingers. From a purely HCI perspective, this 
allows for the natural manipulation of digital constructs as natural objects. 

Secondly, and this is in line with the notion of spatial computing, augmented reality is 
superior to virtual reality in that it retains a computational space of the physical environment 
around the user with the addition of virtual contact. From a fundamental perspective, this 
relates merely to keeping the simulated world grounded in that of the quotidian. In a more 
abstract sense, when considering artificial intelligence and managing it in terms of a spatial 
environment, grounding a virtual model in the physical environment is a veracity check. The 
constructs the machine learning models put forward are checked against the potential lack of 
"reality checking" that could emerge in VR. Reality checking concerns virtual environments, 
merging with artificial intelligence and creating infinitely bespoke and aestheticized worlds 
that create significant cognitive gaps between conventional and virtual reality. 

Much of this writer's theory on the importance of grounding virtual and augmented 


realities in some conventional frame relates to Erin Manning's work on embodied thought. 
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Much of Manning’s theory of mind relates to the intersubjective dynamics of relational 
movement and ‘bodying,' or the relation of the body to the physical environment. (2014) 
Cognitive dissonance arises until that body is entirely contiguous to that of the virtual content 
and its relation to the environment. Therefore, this author argues for using augmented or 
extended reality within a regime of spatial computing as the mise en scene of embodying AI 
in Intelligence Augmenting environments. The cognitive load relieved by the information 
synthesis of the AI engine counterbalances with the grounding/checking with the physical 
environment (veracity) and the necessity of intersubjectively 'bodying' with the environment 
and the virtual constructs. In this intersubjectivity, this author argues for the possibility of an 
Engelbartian intelligence-augmenting scenario. 

However, the problem with artificial intelligence remains in its operational flaws, even 
before it can expand into large environments as a structural milieu. However, from a purely 
gestural point, this author agrees with Hegland regarding needing fundamentally different 
interface regimes to incorporate large learning systems. These new systems must exhibit 
robust veracity and allow fluid integration into intersubjective environmental paradigms. In 
this way, AI might be the foundation of creating spatial computing environments that allow 
the human organism to expand within these environments while remaining grounded in 


objective reality. 
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Matthias MUuller-Prove : On the Eve of the VR 
Paradigm Shift 


1963: Douglas C. Engelbart writes A Conceptual Framework for the Augmentation of Mans 
Intellect. 

Ivan E. Sutherland finishes his thesis Sketchpad - A Man-Machine Graphical Communication 
System! on Sketchpad, the first interactive CAD system on a TX-2 computer. 


1973: Xerox Alto — an experimental personal computer at Xerox PARC that was used to 
explore and design graphical user interfaces (GUI), local networking (Ethernet), laser 
printing (raster fonts) and object oriented programming (Smalltalk). Alan Kay coins the 


phrase, » The best way to predict the future is to invent it.« 


1983: Apple Lisa‘ — second commercial computer with a graphical user interface; after Xerox 
Star in 1981 and the Apple Macintosh in 1984 


1993: The World Wide Web is donated to the public domain. [The only remaining legal copy 
of the certificate is owned by Robert Cailliau. /cf. Interview with Robert Cailliau, 2018'] 


And 2023? 


History is perceived as a sequence of logical steps only though the rearview mirror. The brief 
compilation of computing events form 1963 onwards is no exception. Nonetheless, it is quite 
likely that 2023 will be seen as a significant year because it marks once again a shift in 
computing paradigms. We went from punch cards and command line interfaces (CLIs) to 
GUIs. Since the 1990s desktop windows coexist™ with browser windows to access the Web. 
Mobile touch devices are part of our daily life since the first decade of the millennium — the 
iPhone was introduced in 2007. 

Paradigm shifts in interaction design do not replace the previous modalities. In fact they are 
all used in parallel today. [cf. Back to Childhood — Infantilisation of UI Design, 2011"] 
Depending on the tasks, we still enter text commands at terminal shells or search engine’s 
textfields; we drag file icons around, and touch, swipe and pinch pixels behind glass. Voice 


commands basically swap keyboard input with speech recognition. Due to the emulation of a 
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human conversation, the interaction feels even more natural — until some commands are not 
properly interpreted and the illusion falls apart. Users have to cope with it. The systems 
represent decades of development that are layered on top of each other. Usage is only 
intuitive for a very small scope — on a larger scale metaphors and command structures are far 


form coherent and consistent. 


What’s next? 


2023: AI and VR 
The research lab openAI has ignited an industry hype by launching chatGPT 3.5 last year and 


chatGPT 4 in 2023. Large language models and deep artificial neural networks generate text 
at a level of quality that seemed to be reserved for human abilities until now. By harvesting 
an average of all human writing, AI systems get better to “understand” the intention of their 
users and to respond in a seemingly reasonable and correct fashion. However, algorithmic 
philosophy does not yet provide a framework to describe what is happening in this area. Vint 
Cerf put it this way [Future of Text and AI, 20-Apr-2023°]: »Our problem is that the Turing 
test turned out to be too easy now for the ChatGPT to pass.« It might be too early to provide 


an assessment on the techno-social implications of the new computing capabilities. 


The other technique that eventually reaches the professional and prosumer market in 2023 is 
VR/MR/AR (virtual, mixed and augmented reality; cross reality -XR- is used as an umbrella 
term.) XR systems are under development for quite some time. A year before 1963, Morton 
Heilig has introduced the analog system Sensorama, that enhanced a movie clip with stereo- 


audio, aromas, wind, and vibrations. [interview footage with M.Heilig?] 


The Sword of Damocles“ by Ivan Sutherland et al. (1966) is considered as the first computer- 
based MR prototype: The user was able to inspect a free-floating cube by moving around. His 
perspective was tracked with telescopic arms from the ceiling, hence the nick name. Since 
then various research and development teams aim to provide a satisfying user experience in 
virtual worlds for play and work. Google Glass, Microsoft HoloLens, and Oculus Quest to 


enter the Metaverse are recent products. 


For decades Apple has a track record to shape the industry by its products and services. The 
desktop publishing revolution was fuelled by the Mac. The digital music industry was 
completely disrupted by iPods/iTunes which lead to the iPhone and the AppStore market 
place. Other companies like google, Samsung and Microsoft followed to build smart phones 


and tablets and their own operating systems and app ecosystems. 
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Tomorrow, on June-5, 2023, Apple will introduce its VR headset. Like in desktop-GUI 
computing or mobile touch devices, all other research and development teams will conduct 
competitive analyses on Apple’s design solutions. This will inspire and inform their own road 
maps. All together they will establish a de-facto standard how to interact in XR environments 


by means of head-mounted-displays, gestures and plenty of sensors. 


As mentioned before a paradigm shift in human computer interaction has always just added a 
new layer on top of the established stack of HCI design patterns. Users will continue to 
choose the device and the input mode that best suits their tasks. As a consequence this means 
that data, tools and users will and shall be able to seamlessly migrate between devices and 


systems. 


A user — Eve — might start to take notes on a new idea by recording a voice message to 
herself. Later she opens the list in text-form on her laptop computer and continues to refine 
her concept. In order to incorporate related research and references she might virtually dive 
into the ocean of literature and other media of her field. The 3D info space is arranged in a 
way to make her feel comfortable and safe. It enables her to discover related materials with 
ease. Connecting the dots, drawing conclusions, and enhancing her considerations is a 


computer-augmented thought process. 


It is a desired future to reuse and further improve common interaction modalities for XR — if 


possible and if it makes sense. Let’s take a look at the established interaction areas. 


The desktop metaphor is fundamental to personal computing. For virtual and augmented 
scenarios it has to be replaced by a 3D rooms metaphor. Be reminded, that metaphors in 
computing are not just digital copies of real objects; instead they use familiar concepts and 
add extend the conceptual model to offer functionalities that would otherwise not be 
understandable and usable at all. 

WIMP (an acronym for windows, icons, menus, pointing device): windows, i.e. rectangular 
areas with scrollbars to reveal sections of a much larger canvas... Windows can overlap like 
sheets of paper on a desk. Due to the limited screen real estate mobile devices don’t have 
windows Here, the only window is identical with the screen. Virtual rooms do not have edges 
— everything is projected into a sphere. Stereoscopic sight adds the third dimension of vicinity 
and distance. It has yet to be decided if metaphorical shelves —like the stockroom in The 


Matrix'— offer good access to documents and media files. 


Icons either represent (closed) documents and folders or they provide cues for command 
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buttons or apps. Both aspects are needed in XR. But simply using the graphical language of 
GUIs and mobile devices for VR seems not to be appropriate. New symbolic items that are 


born-3D should be developed and evaluated. 


The use of menus is slower than other command panels like e.g. tool bars or hotkeys. The 
primary benefit of menus is a complete overview on the functional space of an applications. 
The lack of a standard menu structure for apps on mobile devices makes it extremely difficult 
to occasionally use apps. Apart from a few standard buttons there is no familiarity with apps 
that are used infrequently. Obscure interaction gestures remain hidden and cannot be learned 
by climbing through the menu tree. XR will face the same issues until standards and 
conventions will help the users to feel safe and in control in new XR app environments. 

The P in WIMP stands for pointing device. Originally it was a mouse, but today trackpads 
are more common than mice for laptop computers. Another advantage of trackpads is multi- 
finger gesture support. These gestures have been adopted from mobile devices. Pointing and 
multi-finger gestures will also be used for XR interactions. New gestures will involve the 
hand, both hands, and even body gestures like nodding or shrugging the shoulders. Some of 
these new gesture can eventually even be used on other device categories. 

A unique UI element that can be triggered by point and click is the hyperlink. It connects 
two Web pages with each other and teleports the user from a marked piece of text to a 
destination somewhere else on the Web. Magic portals are already used in VR games to 
change levels or virtual scenes quickly. Generic XR hyperlinks might connect various XR 
information environments with each other much like a browser jumps from one webpage to 
another. 

VUIs — voice user interface — is the next interaction modality that evolved from speech-to- 
text dictation systems into voice command systems for mobile, car and home entertainment 
and shopping devices. As soon as the error rate comes down to an acceptable niveau, voice 


commands can be an additional input modality for XR environments as well. 


The blinking i-beam cursor has been invented in the late 1960s. 

Larry Tesler introduced text selecting by dragging the mouse and cut/copy/paste commands 
for the clipboard metaphor at Xerox PARC in the 1970s. Innovation research shows that new 
paradigms have a lead-time of at least a decade and also a life-time of half a century and 
more. This can be explained with The Pace Layer Model by Steward Brand [introduced in his 
book The Clock Of The Long Now: Time and Responsibility, 1999]. Changes in infrastructure 
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are much slower than market cycles of products or the fashion collections of the seasons. 
Media theory says that a new medium will first use and embrace the former medium before a 


new appropriate language for its content formats will evolve. 


XR is in the same situation. Prevailing GUI and gesture/touch and VUI interaction design 
systems will be first adopted for XR before a mainstream of typical interaction behaviours 
gets established. Chances are high that 2023 will be seen as the year of an AI break through 
and of an paradigm shift for XR due to the launch of Apple’s headset. 


It is important to consider the entire suite of devices and to design for scenarios that involve 
the flawless migration between devices, various applications, apps and virtual environments. 
Proprietary silos will slow down innovation. Generic and flexible interaction methods, that 
are always available like the clipboard, will improve the adoption of XR environments and 
increase the joy of use. Standard system behaviour will flatten the learning curves an improve 
productivity. 

Now is the time to shape the XR tools in order to support and augment collective human 


reasoning and collaboration. 


The user Eve will decide whether Apple’s solution fits her needs and matches her 
expectations regarding augmented and virtual reality — or whether the industry needs to 


come up with even better tools to enhance Eve’s digital life. 
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Peter J. Wasilko : Textual Affordances in VR & XR 


We stand on the cusp of a new era in Human-Computer Interaction. In the days and weeks 
ahead a number of new immersive display platforms will be entering the market with new 
headsets arriving from Meta [13] and Apple [14], with rumors of a Samsung, Google, and 
Qualcomm XR Headset collaboration [15] and hits of new Xbox VR hardware [16] in the 


mix. 


Interaction Modes 


Rather than speculating on the specifics of any given current or emerging offering, we can 
abstract away all the details and think in terms of operating modes, some or all of which 


might be supported in any given hardware configuration. 
Full VR 


In a full VR mode, all external audio and visual stimulation would be blocked by the 
hardware to present a user with a fully immersive virtual environment which might take the 
form of a totally abstract data space [17], a freely navigable semi-realistically to photo- 
realistically rendered 3-D Virtual World like SineSpac [18], or a passive 3-D Film or 
navigable set of linked 3-D Photo Nodes in the QuickTime VR / Google Street Views 


tradition. 
Basic AR 


In a basic AR Mode, one’s device either consists of clear lenses off which an auxiliary Heads- 
Up-Display image is projected (rather like the classic Pepper’s Ghost stage illusion) or of a 
traditional VR headset augmented with high resolution pass-through video cameras 
presenting an image of the outside world composited with computer generated imagery to 
superimpose virtual screens on one’s environment. Such actual or virtual HUDs can either 
track one’s head motions to appear fixed relative to it or be dynamically resized and 
reoriented in response to head movements so as to appear to be stably anchored at a fixed 


position in 3-D Space around which the user can freely move. 
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Mixed/Extended XR Mode 


Finally, in an XR Mode, one’d headset is further augmented with LIDAR or other advanced 
sensors allowing the device to generate a 3-D model of the space around it, making it 
possible for the system to identify real world surfaces and planes onto which 3-D Models can 
be overlaid so one might be able to replicate the appearance of the Holographic Chess game 
in Star Wars: A New Hope or to seamlessly replace a wall poster present in real world or 
create the illusion of a real live kitten frolicking on one’s sofa that cannot be visually or 
audibly distinguished from a real pet until the headset is removed. In theory, future 
generations of immersive technology may eventually employ room-sized holographic, light- 
field, or auto-stereoscopic displays; or give way to neural chip implants to obviate the need to 


wear a headset. 


Addressability 


With this broad range of possibilities in mind, let us now turn to the place that Text will have 
in such environments, but first we need to briefly touch upon the critical concept of 
Addressability. 


Spatial Addressability 


First, we need a way to name the Space we are dealing with, which can either be the Real 
World, some arbitrary Benediktine Cyberspace , a small local simulation like The Construct 


in The Matrix, or a massively multiplayer online role playing game universe, like Eve Online. 
Temporal Addressability 


We then need to specify a Time Index which can be an absolute point or range in time at any 
given granularity of concern (e.g. a given year or a specific date), a relative temporal offset 
(e.g. “531 milliseconds’ into a named experiment, ‘5 to 7 seconds’ of a given YouTube video, 
or ’20 minutes after rebooting’), or cyclic (e.g. ‘every third Tuesday’, ‘in the winter’, or ‘after 
dark’). 
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Granularity and Naming 


Similarly, we need to be able to specify regions / sub-spaces and point locations which might 
be given in absolute geospatial global coordinates, as a named geographic region with known 
mapping to such coordinates, or in a relative fashion with reference to a model or model class 
like ‘the kitchen’, through an explicit naming of a 3-D Cursor location, or a spoken or typed 


search query capable of returning multiple results as a list abstraction. 
The Complete Viewspec 


Beyond simple search queries we can also assume that various filters and modalities might be 
employed by the system whose parameters would also need to be tracked along with a 
Viewpoint and Orientation within or relative to the space or point under consideration. Only 
with this full range of data would we be able to specify a Viewspec [19]with sufficient 
information to capture what we are seeing in VR. Note that we are contemplating the ability 
to replicate a navigable immersive experience in toto rather than making a 2-D screenshot of 
the virtual world or creating a static instantaneous 3-D Model of that portion of the VR World 


visible to the user at the moment of requesting the viewspec capture. 
Implementation Possibilities with IPFS and Mnemonic ID's 


In order to achieve this level of full addressability and fidelity, we are essentially 
contemplating the sharing of the full VR Model + Viewspec + data-streams employed by the 
system at the moment of capture. Clearly, this is too much data to be packed into a feasibly 
exchangeable identifier, so the model and data stream would no doubt be incorporated by 
reference to pre-packaged physical media or networked resources. Even so, the balance of the 
viewspec would be too beefy to fit in XR Fragment [20], but we can get around this by 
putting the full viewspec and associated data-stores on the IPFS under a Humane Mnemonic 
Identifier like “Purple Elk on Mt. Fuji” [21]. 

Assuming now that we have a way to select objects and spaces in our virtual 
environment and that we can associate them with Mnemonic ID’s like “Green Crane under 
Willow” we can think about how to work with text in VR. 
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Severability and Transferability 


Some years back I proposed a Principle of Severability [22] arguing that users would be 
unwilling to invest extensively in creating user-generated-content within an any given 
environment unless they were provided with a mechanism to export that content to modify it 
or in a worst case repurpose it should the platform fail. 

Today, I would like to make a stronger claim for a Principle of Transferability and argue 
that for Texts to achieve their maximal utility in a VR/XR World they must be fluidly 
transferable within and across platforms. The closest example of such a capability is offered 
by the MacOS Handoff facility which synchronizes the clipboard and some application state 
across devices sharing an AppleID iCloud login within the Apple Ecosystem. But this level of 
Limited Transferability falls short of supporting an XR Environment that might contain fully 
virtual monitors, representations of Windows, Linux, Mac, and home-brew based devices, as 


well as avatars of participants using multiple-brands of headsets. 
Stephenson's Hypercard 


What we need is something more akin of the Hypercard of Neal Stephenson’s Snow Crash 
[23] which served as an avatar for an arbitrarily large and complex chunk of data. In short, 
Stephenson’s Hypercard is a virtual artifact represented visually as a traditional business card 
holding the address of a datastore that could be copied via a handing off gesture to provide 


the recipient with copy-on-write access to that data. 
Pick and Drop 


Clearly we need a mechanism to transfer text and other data across devices that aren’t 
necessarily owned by the same person unlike the current case with Apple’s Handoff 
technology which doesn’t permit handing off between multiple users. Sony’s Jun Rekimoto 
anticipated this concern in the late 90’s by proposing a pen-based mechanism called Pick and 
Drop [24]. 
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I'm physically picking up 
a computer object with 
my pen! 


I'm transferring computer 
data through the network. 
° 


Figure 2: The conceptual difference between remote copy and Pick-and-Drop 


This system would employ a “pen” artifact that could be correlated with a data object on 
screen, such that the Pen’s PenID would be bound to a network data transfer specification 
allowing a receiving device to automatically initiate a background network file transfer of the 
associated data object. Rekimoto’s system was somewhat limited by the need to construct 


pens whose physical proximity and ID could be detected by target systems. 


Toward Tangible Hypercards 


In VR we can combine Stephenson’s and Rekimoto’s insights to envision a cheap deck of 
laminated playing cards bearing a Mnemonic ID on one face and a QRCode on the opposite 
face. When produced in an XR environment the System would use machine vision to register 
the QRCode and track its position in World. We could then support a familiar ID Badge 
Swipe gesture waving the card over a textual selection or a real or virtual object present in the 
environment to copy it onto the card, this could cause the system to simulate a color change 
to indicate its loaded status. Tapping the card against a data reception target in the XR 
environment would effect a data transfer. In addition to tracking hypercards, the system 
would also be able to register and subsequently track tablets, phones, and laptops by having 
them display a QR Code or post their GPS coordinates. This would permit the environment to 
orchestrate the transfer of data objects like large texts and corpora across physical devices in 
the real world. Moreover once bound to a hypercard, data could be transferred outside the XR 
Environment by carrying the physical card to some other networked device where its QR 
Code could be scanned or by carrying its Mnemonic ID in one’s head to be shared from 


human memory! 
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Wands and Crystal Balls 


Where the Hypercard is an avatar for a chunk of data we can also imagine representations of 
data structures and programs. The most natural representations here would be the Wand as a 
stand-in for a Deque (double-ended queue read “deck”) / clipboard data structure and the 
Crystal Ball as a representation for programs. The wand could be a simple black rod whose 
orientation could be distinguished by cap color or width to allow the XR environment to 
visually distinguish a canonical front and back end. 

Tapping a data transmitting object with either end of the wand would push one item per 
tap onto that end of the wand, tapping a data receiving object would draw off one item per tap 
until the receiving object no longer required additional data. Crystal Balls could represent 
programs that might display or alter some or all of a wands contents. This would serve as an 
embodiment of a Forth like programming language and wands would not need ID’s per se as 


they would be identified by their users. 
VR/XR with a 3-D Mouse 


For sit down VR/XR sessions entailing extensive typing of original text, a 3-D Mouse like 
3Dconnexion’s SpaceMouse® Wireless used off-left of one’s keyboard in conjunction with a 
conventional mouse off-right will likely provide a more ergonomic solution than handheld 
controllers. The trick will be to provide a Snap-To facility to allow a user to make a fast 
semantic selection of the nearest contextually appropriate element in the 3-D environment. 
This sort of interaction would fit in perfectly with the sorts of abstract textual visualizations 
explored at the MIT Media lab by David Small whose magna opus Rethinking the Book 
[25]is a must read. See also: [26] [27] [28] 


Purely Gestural Techniques 


Since Mnemonic ID’s can obviate the need to carry around jump drives and scroll url’s on 
the backs of napkins, and since the next generation of headsets will no doubt support noise 
canceling earphones, speech recognition, and high resolution hand tracking, we can 


contemplate a range of approaches that don’t depend on secondary input gadgets. 
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Frame and Dictate 


The first and most critical element to leveraging text in VR is to provide a mechanism for 
rapid text entry. While nothing is likely faster than a chorded Stenography Keyboard or at a 
slower pace a conventional keyboard for a Sit Down work session; when standing, a Frame 
and Dictate methodology would be ideal. Essentially, the user would first create a virtual 
monitor by lifting both arms and touching both pointer fingers together at the top center point 
on the desired monitor and then sweeping her finger tips out the sides, then down, and then 
back together to trace out of the frame of the desired monitor which would then fade in to fill 
the screen with a blinking cursor. The left third of the virtual display would then being to fill 
in with a speech to text transcript of anything being spoken in the room while noise 
cancelation would block those sounds. The user would then dictate text directed to the screen 
while the noise cancelation tech prevent other users from hearing what was being dictated. A 
snap of the fingers would toggle between dictation mode and conversational mode to allow 
one to alternative between dictation and being heard by colleagues until a double click would 


terminate the dictation session. 


Frame and Flatten / Swipe to Select 


Another key affordance is the ability to detect and capture text in the environment. For this a 
Frame and Flatten gesture could be provided where one would raise one’s hand with thumbs 
and fingers up at 90 degrees to form a virtual text capture frame that could be scaled, and 
spun in 3-space by changing the spacing and orientation of one’s hands. Once in a workable 
scale and orientation a two finger swipe could be employed to select and highlight or capture 


text. 


Ant Man Scaling Effects 


In a truly advanced XR environment, pass-though video would be replaced with real time 
rendering of a Lidar generated model of the space. This would make it possible to play with 
an Avatar’s Scale by reaching out and grabbing mid air to expand or contract one’s scale 
within the space by pulling or pushing the space apart, (i.e. grab in the air and pull your hands 
together to grow your avatar and shrink the space or push your hands apart to shrink your 
avatar and zoom into the space). This affordance would play very well with abstract data 


visualization, but could also help trouble shoot industrial systems where the XR model might 
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be augmented with real world telemetry data and textual documentation panes. While 
zooming in or out, a synthetic shadow in the room could indicate one’s physical location, 
while one’s avatar would appear to grow or shrink by the zoom factor it was operating under. 
Imagine several lawyers studying a contract with one shrinking to mini-me size while 


studying some small print text. 


Tangible Citations 


A natural extension of this sort of visualization would be a document room with multiple 
virtual displays each holding a different legal case or virtual representation of a tangible piece 
of evidence. Implicit in this document collection would be a large number of linkages which, 
if simultaneously visualized, would render the data-space incomprehensible. However, in 
VR/XR we could manifest these links as glowing threads connecting text selections across 
virtual documents only when an endpoint was actively selected. To get a feel for how 
powerful this technique can be in 2-D, play around with the Map view of a series of related 
definitions in Frode Hegland’s Author. 


Implode to Text Effects 


Another closely related use-case might be standing a library or high information density 
environment with lots of things at different scale competing for one’s attention. Here an 
Implode to Text Effect would crossfade from XR to Pure VR fading out everything except the 
text in one’s field of view while animating, rotating, and warping each textual surface to 
bring them into position at a uniform scale in a single space filling two-dimensional layout on 
an ergonomic arc shaped overview panel. (This would also normalize the orientation of 
perpendicular blocks of text on the spine of a book.) The closest cinematic depiction of this 
sort of effect can be seen in some of the visual sequences depicting the operation of Cerebro 
in the X-Men film franchise. 


Pluck and Plop 


Finally at a more prosaic and minimalist level we could employ a gesture based analog to 
Pick and Drop to transfer text between physical devices and user interface elements. This 
basic Pluck and Plop gesture would consist of reaching forward into 3-D space spreading and 


closing one’s thumb and pointer finger as if plucking a string to symbolize the desire to 
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capture the nearest textual or object referent in line with the motion. Where that reference is a 
text pane with some previously selected text the motion would capture both the selection and 
Visual Meta reflecting its source and authorship. Reversing the motion as if tossing a pea 
held between one’s thumb and forefinger would send the data object outward in 3-D space 
until it plopped on the nearest drop target which could be an XR-Representation of an actual 
device; a real or virtual Hypercard; another Avatar who could catch it with a corresponding 
Plucking gesture, in the absence of which, a link to the data would be sent to that user via 
email; a Wand, Crystal Ball or similar virtual artifact; or even oneself to email oneself a 


download link to access the data outside of the system. 


Viewspec Reification — Clap to Capture 


Just as important as it is to be able to capture text, selections, and highlights, we need to be 
able to preserve the State of the Experience. For example in an XR Enabled library, one 
might have employed a sophisticated voice query to highlight books on a given topic that had 
been cited by one of seven authors of interest. Since there is no obvious object, real or virtual 
within the scene to represent this Viewspec, we need a way to reify it into a manipulable 
form. This could be achieved by a Clapping Gesture, which would cause a glowing ball with 
a Mnemonic ID call-out caption to be manifested between one’s hands that would hold the 


viewspec and be subject to Pluck and Plop data transfer. 


Summary 


Regardless of which, if any, of the aforementioned possibilities or variants there of, that see 
actual implementation, we need to find ways to fluidly enter large amounts of text into VR & 
XR and to move them across vendor platforms. Moreover, abstract text and data 
manipulation functions like key word highlighting, named entity extraction, parsing and 
formatting, that aren’t inherently tied to the mechanics of any given platform need to be 
Severable and Transferable across systems so that they won’t be subject to Walled Garden 
Capture. Only then can we see the same level of Openness and Interoperability emerge for 
VR and XR that made the World Wide Web the Civilizational Change Agent that it has 


become. 
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Phil Gooch : Cross hatched 


“Will you take me there 
To a distant place I’ve never been before? 
I could leave this world 
I could follow you like oceans to the shore 
You could take me there 
Make the rivers of my mind flow to my dreams” 


Nitin Sawhney, Immigrant 


Ironic isn’t it? We exist in four-dimensional spacetime, but we increasingly lived our lives on 
a two-dimensional plane. Heads down, swiping, tapping, gazing. Recording live events 
through our phone screens, rather than experiencing the moment. 

Before, when we used to laugh at those early bricks, we couldn’t imagine then how 
these devices would evolve and shape us. How we’d create new vocabularies, new 
connections, new roles. And yet they would limit us. We stopped looking up. 

Later, we chuckled at the strange headgear, the odd gestures. Like silent disco, it was fun for 
a while. But we were still stuck in these experiential silos. 

In The City and The City [29], China Miévillegave us a glimpse of what was to come. 
While the citizens of BesZel and Ul Qoma had to learn to un-see (‘When in Beszel, see 
BeszZel’), we embraced the cross-hatching of boundless realities. Just as three-dimensional 
space stacks an infinity of two-dimensional planes, so we learned to tesseract worlds. 
Remember the big reveal? Those extended earbuds that looked like nothing special. But the 
holograms they projected amazed us. The silos were gone, we could overlay new realities 
seamlessly. We could issue commands by thought, and the AI would turn these into new 
environments. There was no ‘virtual reality’, instead, we could create multiple actualities. 
And not just on Earth. 

Remember the first trip to Mars? We’d had the robots up there for a while, they had 
built the tools they needed to reproduce themselves and 3-D print the structures. But making 
the physical journey ourselves still eluded us. Instead, we used the augmented perception 
technology we had here to transmit and connect our senses with the machines there. Sure, the 


ten-minute time delay is a bit of a pain - but that’s a temporary limitation. 
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We may never be able to send matter through a wormhole, but sending information might just 
be possible [30]. 
If all our sensory information can be encoded in bytes, what really makes us human? I 


guess that’s a question for another time ... 
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Rollo Carpenter : Versality, Chapter 2.09 


Though possibly it was ill-advised, Emi was flattered to be invited to meet Xeen-Xeenic, in a 
non.life space to be opened the next day — just the two of them, ‘on camera’. A very effective 
promo for the space-designers it no doubt would be too. 

“How are you on a surfboard?” asked Xeen in-ear, while Emi was at work. You was on 
the asterdroid Targ, in the middle of attempting and largely failing to make humorous 
conversation with a gaggle of low, squelchy beings around a sulphurous pool. 

“Surfing? Well, you know, I’ve been in the sim-spaces with friends several times... but 
they’re never quite right.” 

“This place is different. Will send the link.” 


Space: NL-ALT1933 SEA-SPACE1.0 
Time: 2098-06-18 15:15:15 gmt 


Once blipped, there they were, barefoot in full wetsuits, on a warm broad curving sandy 
beach, boards in hand, pointing skywards. Not quite a normal beach — not in geometry, scale 
or light. Somehow, without quite seeming too overwhelming, behind them stretched most of 
the African continent, with the Elevator visible on the horizon. A glow beyond, within, lit all 
of it, but the sky was a deep moonless night, stars and Milky Way in brilliant detail, closer 
and more vivid than life, and rotating visibly fast, as the small world turned. 

With their backs to the sea, it was almost above them. As they turned to the sea, now 
the land was above them, and the sea an approachable surface, full of dramatic glowing 
waves. The water was as if in sunny daylight, and yet, magically semi-transparent, the 
strongest of stars shining right through it, well below the ocean’s peculiarly infinite and non- 


curving horizon. White crests atop the waves were spacily — almost darkly — translucent. 
“This looks pretty fantastic, Xeen. So we surf here, now?” 


“Ooh no, Emi, we go places. Surf places. You’ll see! First we have to get out through 
the breakers.” 


Boards lowered to their sides, they ran into the waves. As they paddled themselves out, 
dipping under crests, Emi called excitedly to Xeen, close by, “This is no normal non.life 


water. This is the real thing!” 
“Yes. Problem now solved. But look behind for a moment.” 


As they moved away from it, the beach, Africa — the world was becoming a globe. Soon 
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enough, it was complete — a jewel-like Earthly sphere was floating there in the broad ocean, 
two fifths still slightly visible beneath the surface, turning. The beautiful night sky above 
them no longer appeared to be moving. They were in the deep — in deep sea space. 

Xeen spoke again, perfectly clearly despite the waves regularly crashing over them, 
“The creators of this space probably expect everyone to go outwards first, via Mars and 
especially Jupiter. So let’s go inwards.” Xeen selected your intentions, in-eye. 

The sea adjusted beneath them, and the real waves came, luminously, in the day-night. 
At the third attempt Emi and Xeen both caught their first, only a few metres apart from one 
another. This was no mere coincidence. They did not only ride the waves. The waves rode 
them. 

While ensuring it was never too easy, this space could understand human actions and 
micro-configure the water to ‘happen to’ arrive just right to ensure that even whole groups of 
friends stayed close enough together, caught pockets and wiped out at nearly the same times. 

Xeen knew the Sea-Space was built for maximum visual impact too. If they aimed to 
surf straight to Mercury, Venus was bound to be conveniently arranged to pass close by. In 
the event, it came more than close. 

Just as Emi and Xeen whooped with joy at catching the second of a set of waves, and 
were travelling at what to scale must be near light speed, the hazy light of Venus loomed up 
rapidly on their right. 

“Let’s try not to collide with that thing, eh Emi?” Xeen said. 

Emi raised her eyebrows and smiled lopsidedly at Xeen for a second, but did not have 
much of a chance to respond to the jibe. The invented, hyperbolic orbit of Venus sent it 
sweeping rapidly by, right in front of them. It cut a dramatic curving path through the waves, 
creating complex disruptions in the surf, arriving at drastically unexpected angles. Wipeout. 

Yet, a mere five caught waves later, they approached Mercury, elated but already 
grateful for a new gentleness of the water, carrying them easily towards its ‘shore’ as the 
whole planet distorted in their vision, down, around — from sphere to surface. 

They raced out of the waves, finding Mercury unmysteriously hot and hard to their 
barefoot touch, laughing with exhilaration and at the sheer absurdity of it all. 

“Next stop, the Sun!” shouted Xeen. 

“But the sea will boil,” laughed Emi. 

They could have walked all the way around Mercury, but before long jumped right back 
in. The sea did boil. Xeen and Emi landed ‘on’ the tumultuous Sun in huge clouds of steam, 
and with great shrieks — but their wetsuits, skin and eyes did prove, thankfully, suitably 


resilient. 
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“Don’t tell me... onwards to a hypothetical interpretation of Gliese 667C and its 
planets!” Emi said, humorously hopping from foot to foot, in a pointless yet seemingly 


unavoidable attempt to improve matters a fraction. 


“Why, Emi... how did you guess?” Xeen replied, voice raised above the noise of a 
nearby erupting solar flare, while gesturing open an intensely-glowing screen, to make their 
more complex, abnormal selection. 

They paddled away from the surface of the Sun, now a vast brilliant globe, towering 
above them, filling most of the sky, floating there in the infinite ocean universe. For a 
moment the heavens rotated, all for them, so as to be on the correct axis for the destination 
star. 

The first wave, which was approaching to transport Emi and Xeen unimaginable 
interstellar distances, at imaginarily many times the speed of light, arrived, so tall that it 
almost engulfed the Sun. Within seconds of catching it, the Sun was Sol — far behind them, 


and just another unremarkable point of light in the sky. 
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Sam Winston 
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Vint Cerf : On Virtual and Augmented Reality 


VR and AR have been the ‘coming technology’ for some time now. 

A major challenge has always been comfort or discomfort of wearing a headset for long 
periods of time. For headsets to work in AR, RR (real reality) needs to come through and be 
visible in high resolution in addition to any augmentation. Interactions in AR/VR have also 
been challenging. Simulated hands and tools have been made to work but without haptic 
feedback, they are less intuitive. 

Holographic displays would be a potential way around the usual headset but often have 
limited applicability for groups since the focus may only be optimal for parties at a particular 
focal point. 

Direct manipulation is more intuitive and attractive when this is possible. As we have 
learned with chatbots, application programming interfaces that allow the bot to interact with 
the real world through the virtual one is empowering. This could extend to direct control over 
‘Internet of Things’ devices and general software applications. 

VR environments often need avatars to facilitate interaction among participants. 

One wonders to what extent avatars should resemble the real users? Of if they do not, 
whether some kind of warning is needed? 

Should there be any kind of validation of an avatar/person correspondence? 

What about ‘made up’ people whose faces look real but are artificially generated? 

Should they be explicitly distinguishable from real people? 

How might that be enforced? 

What standards of identity should apply? 

If chatbot helpers are created, should they have ‘verifiable pseud-identities’ so they are 
NOT confused with real people? 

Will companies using artificial assistants be compelled to identify them as such? 

Is there some measurable threshold after which an artificial bot should be treated as 
having any kind of rights? 

Would it matter whether they were corporeal or not? 

We have barely scratched the surface of a new phenomenon and its implications. 


Where will we be in ten year? twenty-five? fifty? 
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Fabien Benetou & Vint Cerf at The Future of Text 2022. Hegland, 2022. 
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Frode Hegland : Coda. Towards a more colourful 


text 


At the time of writing this, I am waiting for a new version of Reader to test a new way to 
look through a document. The issue is that text is great for linear reading and also great 
in a graph or map view, but not very good for representing overviews‘. For something 
like our books and journal, we need a more dynamic and non-linear way for the user to be 
able to get clued in to what different articles are about. 

In Reader you can already perform ‘Find’ commands which results in only showing the 
lines of text with the search term, as well as keyboard shortcuts to show only lines with 
names or lines with glossary terms. This is useful but still not very ‘glanceable’ and the text 
requires reading to see what’s where. Another interaction which is currently possible and 
useful is that with PDF documents which have Visual-Meta there is also a different way to 
thumb through the pages, letting you use right arrow key for next page but down arrow key to 
go to the next article, which has turned out to be very useful in going through articles at pace, 
without having to spend mental effort skipping pages to go to the next article. But anyway, 


back to the colourful point. It is a simple affordance: 


e The user has a keyword colour-dictionary (initially defined by the producer of the 
software—me—but fully editable by the user) where sets of simple keywords are grouped to 
have specific colours. For example, anything relating to VR is red while anything related 
to AI is dark green. Knowledge work tools and techniques are in dark orange/brown and 
architecture terms in orange. 

e When the user fold the document into an outline showing all the headings in the 
document, any text that is in the keyword colour-dictionary appears in the preset 


colour. 


Note that this only happens when the user folds the document, the colours are not visible 


when reading normally. 
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We are experimenting with also letting the user access this view through cmd-shift-k (for 
Keyword) but initially I think folding is the way to go. 

The possible set of colours is larger than a person can reasonably distinguish without 
seeing the colours side by side (dark blue, light blue and come to mind). Some of the colours 
are not easy to read on a white background (yellow and pink for example). But this slightly 
beside the point. 

The expectation is that a user could choose to focus on one or two colours to start with, 
such as red and green, as mentioned above (red for VR and green for AI). I don’t think it will 
take long for that user to start to associate the colours with the themes so that when they fold 
into the outline they can get the gist of what articles primarily cover what content. After all, 
an article discussing architecture in VR would show a series of red and orange, while an 
article discussing AI and knowledge work would have a series of keywords under the heading 
in dark green and brown. 

Doug Engelbart told me to try colour coding in this way many, many years ago and I 
could never make it pleasant, since reading in colour is distracting to me, but basic tests 
showed the concept made sense. What I think I have done here is providing a way to have 
a useful colour based overview while not at all intruding on the reading experience. 

The next step, if this turns out to make sense, is to try further views, or ‘ViewSpec’s as 
Doug used to call it, to help the reader navigate and browse large compendium documents, 
primarily. I am a self funded developer with a very small budget and that may have been a 
blessing in this case since I needed to use as much of the components we already have to 
build this, and that helps the user as well, since the interaction becomes more integrated into 
the current work practices in Reader. It will be interesting to see. 

So I am writing this day day before Apple introduces their Headset, sitting in the 
bottom of the garden on a sunny Sunday, typing away on my (relative to VR) tiny 13” screen. 
I don’t mind this screen for typing, but the 27” Studio Display in my office is a different 
world when editing. The dream, for me, is to be able to work on either, and when I really 
need to think, put on a headset and fade everything but my computer screen, which I might 
grab and pull up to be a bit taller) and write. VR should, initially, be a powerful extension of 
the work environment we already use, and will likely continue to use. But when I put on a 
headset to think and concentrate, I want to be able to use the full environment, including 
such colour coding as mentioned here, to have a much more powerful, glanceable 
environment outside the ‘main display’, with what I am working on and my resources 


rendered in useful forms in the background, to the sides and so on. 
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The power of VR will be to escape information flatland when desired, but it will be key to 
work in any mode I want. This is why I developed Visual-Meta, though the colours should be 
for the reader to decide. 

Tomorrow IIl see Apple’s headset, Tuesday I expect to be able to test the keyword 
colour-dictionary and Wednesday there is an upgrade to my favourite PlayStation game, 
Battlefield, where an immense explosion of interactivity is possible, where I can run and gun 
and fly and drive and be a minor superhero. Will we be able to make working in VR as 
exciting, will we be able to make working in VR even more richly interactive. I most 
sincerely hope so, but for that to happen we will need to dream, experiment to experience and 


keep questioning, which is exactly what the Future Text Lab community will continue to do. 


Join us, if you are not already aboard: https://futuretextlab.info 
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Glossary 


academic someone who reads academic documents, usually associated with an academic institution, 


but not necessarily. 


ACM From Wikipedia: “The Association for Computing Machinery (ACM) is a US-based 
international learned society for computing. It was founded in 1947 and is the world's largest 
scientific and educational computing society.” 

https://en. wikipedia.org/wiki/Association_for Computing Machinery 


ACM’s ‘Hypertext’ Conference piloted Visual-Meta in 2021. 


Addressability refers to how something can refer to something else. In the digital world this 
primarily means in a local file structure and a server based network. In academia is can refer to 
citation or references. In the physical world it can refer to a formal location layout, such as a flat so 
and so in house so and so, as well as co-ordinates. It can also man relative addressing, such as saying 


take a left after the third yellow house. 


AGI (Artificial General Intelligence). AI stage 2. 

If we are to see a world like the movies with artificial intelligence, AGI will need to come into full 
fruition. AGI would mean a machine would be capable of understanding the world as well as any 
human. All while maintaining the same capacity to learn how to carry out a huge range of tasks. The 
reality? AGI is not yet among us. It is a hypothetical form of AI. The potential is there but some 
believe it to be decades away! 

According to a study by Muller and Vincent C., the top-cited 100 AI researchers have a collective 
90% confidence that AGI will be among us by 2070. To determine if a machine is truly AGI, it has to 
pass the tests: 

- Turing Test 

In this test, 2 humans and a machine look at something they haven’t seen and converse about it. The 
conversation is recorded and an evaluator determines which descriptor is the machine. The machine is 


only disqualified if the evaluator is positive it can identify it. (check out Touring Test on Wikipedia ) 


- The Coffee Test (Wozniak) 

If AGI machines need to understand the world as well as humans do, they must know how to make a 
good coffee. In this test an ai robot is required to go into an average home and figure out how to make 
coffee. It involves finding the coffee machine and adding coffee, as well as sourcing a mug and 
correctly brewing the hot drink. 

- Robot College Student Test 

A literal test! the machine would take the same university classes as humans and complete the exam. 
Some may argue that the machine is cheating with its built-in search engines but who is going to 
argue with a robot... 

- Employment Test 

This is where we should begin to worry about an I-Robot world. This test determines whether a 


machine can perform at least as well as humans in the same jobs! 


AI ‘Artificial intelligence’ is “intelligence demonstrated by machines, as opposed to the natural 
intelligence displayed by animals including humans.” https://en. wikipedia. org/wiki/ 


Artificial_intelligence 


#Technology | Timeline: 20th century- 


ANI (Artificial Narrow Intelligence) 


Al Stage 1. 


Often called Weak AI, this is the AI already embedded in our daily lives. A nice way to describe it is 
that ANI is super-human at specific singular tasks. Like teaching an AI machine to play chess! and it 
will master it beyond human potential with incredible consistency. Challenge it in draughts though? 
and you’ll get a much needed ego boost. This machine can only play chess. Without prior training and 
programming in draughts, it doesn’t stand a chance. Think of AI as hundreds of interns capable of 


highly specific tasks more efficiently than humans. 


ANI has already transformed the way we live, but it has its limitations. A lot of you will already 
interact with ANI on a daily basis. Think of talking with Siri or Alexa on your smart devices, or facial 


recognition when opening your phone! 


Annie Murphy Paul From an interview in GQ: “We make better use of our cognitive resources, says 
Paul, when we use them in conjunction with “extra-neural” resources: our body (embodied cognition), 
our environment (situated cognition), and the people around us (distributed cognition). The brain 
evolved to move the body, to navigate through space, to interact with other people,” says Paul. 

“Those are these human strengths that we're totally putting aside when we focus on the brain and we 
think, ‘To get real thinking and real work done, I have to sit still, not talk to anybody, and just push 
my brain harder and harder.’ It's just not working very well.” 
https://www.gq.com/story/extended-mind-annie-murphy-paul 


Author of The Extended Mind. 


Timeline: -. 


AR ‘Augmented Reality’ is an interactive experience that combines the real world and computer- 
generated content. The content can span multiple sensory modalities, including visual, auditory, 


haptic, somatosensory and olfactory. (Wikipedia). 


Included in the term XR. 


AR includes the real reality, as opposed to VR which does not. 


“Augmented reality (AR) is an interactive experience of a real-world environment where the objects 
that reside in the real world are enhanced by computer-generated perceptual information, sometimes 
across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.” 
https://en. wikipedia.org/wiki/Augmented_reality 


#Technology | Timeline: 20th century- 


AR “Augmented reality (AR) is an interactive experience of a real-world environment where the 
objects that reside in the real world are enhanced by computer-generated perceptual information, 
sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and 


olfactory.” 


https://en.wikipedia.org/wiki/Augmented_reality 


AR includes the real reality, as opposed to VR which does not. 


ARC Doug Engelbart Organisation: Augmentation Research Center, The name of Doug's lab at SRI 
where he proposed a system called H-LAM/T in 1962 and developed and in 1968 demonstrated 
NLS: oNLine System, his platform for shared knowledge work research, later renamed Augment and 


from which I decided on the name Author, since Author and Augment share etymological roots. 


Article A published paper, in a Journal, for academic workflows. 


Artificial General Intelligence Stage 5. 


(AGI) This is the main objective of many scientists working in the field of AI- to develop a machine 
with intelligence like humans. The artificial general intelligence or self-aware systems are not in 
application to date but not many years left to discover this avenue. This is the stage represented in 


many sci-fiction movies that machines are leading humans in intelligence. 


ASI (Artificial Super Intelligence) 
Stage 3 AI. 


Just when you think AI has reached its limit, we bring you the AI of all AI. Super-Intelligence is quite 
self-explanatory. Oxford Philosopher Nick Bostrom describes ASI as “any intellect that greatly 
exceeds the cognitive performance of humans in virtually all domains of interest”. Again, this is a 


hypothetical agent and one which is decades away. 


ASI has huge potential but it also has huge risk. Stephen Hawkings feared its development could be 
the worst event in the history of our civilisation. If we create superintelligent machines we may put 


our own existence at risk, but this is just the plot for another sci-fi film. 


The reality is that AI is a tool for humans to better our existence. Advancements in artificial 


intelligence mean the sky’s the limit! The question is — will the limit bring more positives or 


negatives? We must tread carefully with AI whilst simultaneously embracing it. 


Augmented Text Company The company which produces these tools: Author, Reader and Liquid. 
Director is Frode Hegland. 


https://www.augmentedtext.info 


Author means in the context of this work someone who writes a document, which includes academic 
papers and books. 

Etymology: “mid-14c., auctor, autour, autor “father, creator, one who brings about, one who makes or 
creates” someone or something, from Old French auctor, acteor “author, originator, creator, instigator” 
(12c., Modern French auteur) and directly from Latin auctor “promoter, producer, father, progenitor; 
builder, founder; trustworthy writer, authority; historian; performer, doer; responsible person, 
teacher,” literally “one who causes to grow,” agent noun from auctus, past participle of augere “to 
increase,” from PIE root *aug- (1) “to increase.””’ From https://www.etymonline.com/search? 
q=Author 

Category: Software 


Timeline: - 


Can also refer to the Author software by The Augmented Text Company. 


Author is my word processor for macOS, maybe later for iOS, which produces PDF’s with Visual- 
Meta for any PDF viewer but which my Reader PDF viewer can parse to give the user rich 
interactions. It is produced by my small, independent software development company The Augmented 


Text Company LTD, with programming by Jacob Hazelgrove. 


https://www.augmentedtext.info 


Barbara Tversky Guest presenter at the Future Text Lab. 
Issue 1.1 on the 21st of January 2022. 


Professor emerita of psychology at Stanford University and a professor of psychology and education 


at Teachers College, Columbia University.Tversky specialises in cognitive psychology. 


Author of Mind in Motion: How Action Shapes Thought. Basic Books, 2019. 


Benediktine Cyberspace A 3-D visualization inspired by Michael Benedikt’s seminal text, 


“Cyberspace: First Steps” 


Book a published work, on any substrate, inclduing paper or digital. 


Generally longer than a ‘paper’ but in terms of digital work, there is little technical difference, the 


difference is mostly social. 


Bootstrapping Doug Engelbart concept.: The act of co-evolving the tool and human systems to make 
better tools and systems, thus pulling us up by our bootstraps. (For an overview, see the Bootstrap 


Paradigm Map). This is done on three levels of activity. 


Christopher Gutteridge is also a contributor to the first volume of The Future of Text. 
Lives: Southampton, UK. 


Works at the University of Southampton. 


Citations In this paper, the term’ citation’ is both the ‘citation’ mention in the body of the document, 
and its corresponding ‘references’ entry for the source material in the Reference Appendix at the end 
of the document, though for the sake of clarity I will mainly use ‘citation’ to refer to the in-body half 


and ‘reference’ for the entry of the source in the References section. 


Cognition “is the mental action or process of acquiring knowledge and understanding through 
thought, experience, and the senses”. 


https://en. wikipedia.org/wiki/Cognition 


concept, defined concept, defined concepts Means, in this context, my experiment with useful units 


of thought or knowledge, which can be defined in Author, hopefully soon in Reader. Stored as text. 
The Map view uses the definition to draw lines where text from a definition is also present on the 
Map. 

When a document is exported to PDF the Defined Concepts become Glossary Terms. 

This is as opposed to inferred concept. 


Also aView in Author and in Reader (with Visual-Meta) to show all named entities (and headings). 


Cyberspace “The term "cyberspace" first appeared in the visual arts in the late 1960s, when Danish 
artist Susanne Ussing (1940-1998) and her partner architect Carsten Hoff (b. 1934) constituted 
themselves as Atelier Cyberspace.” Used in this context to mean a digital world, not necessarily in VR 


or AR. 


David Johnson Co-author of Law and Borders, The Rise of Law in Cyberspace 


Deena Larsen is a new media and hypertext author involved in the creative electronic writing 


community since the 1980s. From USA. 
https://en.wikipedia.org/wiki/Deena_Larsen 


Timeline: 1964- 


Definition when it comes to glossaries, it is the user’s stated meaning of a term. 


Document Can be any type of encapsulated information on a computer. 


DOI Document Object Identifiers. An effort to make addressing academic documents via the web 


more robust. 


Used in Author to let the user paste a DOI to cite an academic document which is then sent to 


CrossRef to be parsed into BibTeX which is then used to create a full citation. 


Doug Engelbart, Douglas Carl Engelbart, Doug “He was an engineer and inventor, and an early 


computer and Internet pioneer. He is best known for his work on founding the field of human— 
computer interaction, particularly while at his Augmentation Research Center Lab in SRI 
International, which resulted in creation of the computer mouse, and the development of hypertext, 
networked computers, and precursors to graphical user interfaces. These were demonstrated at The 
Mother of All Demos in 1968. Engelbart's law, the observation that the intrinsic rate of human 
performance is exponential, is named after him.” 

https://en.wikipedia.org/wiki/Douglas_Engelbart 

He was also my mentor and greatly influenced my work, resulting in my company called The 
Augmented Text Company and my word processor being called Author, in honour of his ‘Augment’ 


system. Visual-Meta is inspired by his Open Hyperdocument work. 


Doug Engelbart “He was an engineer and inventor, and an early computer and Internet pioneer. He is 
best known for his work on founding the field of human-computer interaction, particularly while at 
his Augmentation Research Center Lab in SRI International, which resulted in creation of the 
computer mouse, and the development of hypertext, networked computers, and precursors to 
graphical user interfaces. These were demonstrated at The Mother of All Demos in 1968. Engelbart's 


> 


law, the observation that the intrinsic rate of human performance is exponential, is named after him.’ 
https://en.wikipedia.org/wiki/Douglas_Engelbart 


He was also my mentor and greatly influened my work, resulting in my company called The 
Augmented Text Compant and my word processor being called Author, in honour of his ‘Augment’ 


system. Visual-Meta is inspired by his Open Hyperdocument work. 


Egypt https://en.wikipedia.org/wiki/Egypt 


#Location 


Fabien Benetou Prototyping - European Parliament Innovation lab WebXR consultant - Former 


UNICEF Innovation Fund WebXR technical advisor. 
https://twitter.com/utopiah 


https://fabien.benetou.fr 


Find View in Author and in Reader (with Visual-Meta) where the user can see only the senteces with 


keyword. 


flatland a semi-humorous term we use to refer to traditional displays, as opposed to the augmented 
environments of VR and AR. We feel that it is important to be able to move data, including metadata 


between these environments. 


Folded View in Author and in Reader (with Visual-Meta) where the user can fold the document into 


an outline. 


Folding Folding of a document into a table of contents is enabled through Visual-Meta. 


Frode Hegland Director of the Future Text Lab (FTL) https://futuretextlab.info 


Director of The Augmented Text Company where he designed the macOS Author word processor, 


Reader PDF viewer and the Liquid text interaction tool. 
https://www.augmentedtext.info/ 


Editor of the 'The Future of Text’ series of books and Journal, which will be the basis of future 'The 


Future of Text' volumes. 
Hosts 'The Future of Text Symposium! annually, starting 2011. 
Designed Visual-Meta. 


His mentor was Doug Engelbart and influenced by Ted Nelson. 


Future Text Lab A lab dedicated to studying how working with knowledge, particularly text, can be 
done effectively in VR/AR/Metaverse. We host Guest presenters and publish a Journal which is part 


of The Future of Text series. 


https://futuretextlab.info 


Gesture Interaction through hands and body. 


glossary A glossary in this context means a local definition, by the author or editor of the document. 


Glossary means, in the context of this work, terms and definitions in a ‘Glossary’ Appendix in a 


document. 

This can be created in Author as a Defined Term which is then exported as a Glossary and included in 
Visual-Meta. 

Reader can parse the glossary terms in the Visual-Meta and make them interactive. 


This is different from a dictionary since dictionary definitions have general validity and Glossary 


terms only aim to express the author’s point of view. 
Defined Concept becomes Glossary Term when exported from Author to PDF. 
Inspired by discussions with Doug Engelbart. 


A form of metadata. 


Glossary In the context of my work and thinking, a glossary is a set of Defined Concepts which an 
author has created and which is then exported as a Glossary for the reader. 

The primary purpose of this is for the author to have to think through their writing by explicitly 
stating what something is. The definition of the defined concept can then include text which also has a 
definition and in the Map view in Author this connection can be shown as a line when either defined 
term is selected. 

The secondary purpose is for the author to edit this Glossary to make sure it is coherent for export and 


it can then help a reader understand the author’s intentions. 


Google “is an American multinational technology company that focuses on search engine technology, 
online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial 


intelligence, and consumer electronics.” https://en.wikipedia.org/wiki/Google 


Timeline: 1998- . Location: USA 


GPT “Generative Pre-trained Transformer” inclduinig GPT-2 and GPT-3. 


https://en. wikipedia.org/wiki/GPT-2 
https://en. wikipedia. org/wiki/GPT-3 
Product of OpenAI. 


HMD Head Mounted Display for use in VR or AR. 


HTML “The HyperText Markup Language or HTML is the standard markup language for documents 
designed to be displayed in a web browser.” https://en.wikipedia.org/wiki/HTML 


A technology we are also using, along with PDF. 


hypertext a term invented by Ted Nelson for interactive and connected digital text. Today it is very 


much about clicking on links but in Ted’s early work it was all about freedom for the user. 


Hypertext a term invented by Ted Nelson for interactive and connected digital text. Today it is very 


much about clicking on links but in Ted’s early work it was all about freedom for the user. 


Immersed Immersed is a virtual reality product, working productivity software for virtual offices. 


https://immersed.com 


Intelligence Augmentation Doug Engelbart presented the notion of augmenting human intellect in 


his seminal 1962 paper, which would be the basis for his ‘mother of all demos’ in 1968: 


“We need to improve how we augment a group’s (small, large, internal, global etc.) capability to 
approach urgent, complex problems to gain more rapid and better comprehension (which can be 
defined as more thorough and more critical) which result in speedier and better solutions (more 

contextual, longer lasting, cheaper, more equitable etc.). And furthermore, we must improve our 


improvement process (as individuals and groups).” 


Doug Engelbart (Engelbart, 1962) 


Journal A collection of articles. For academic use, usually the record of the work of society, such as 


any of the many special interest groups at ACM. 


Journal The monthly Journal of the Future Text Lab. Inspired by Doug Engelbart’s journal and 
academic journals in general. Published as PDF with Visual-Meta and in .liquid and soon HTML on 


https://futuretextpublishing.com/journal/ 


Leon van Kamme Leon is a Dutch audiovisual prototyper and researcher currently based in 
Budapest, Hungary. 

He is currently working on XR Fragments, in which he strives to bridge his diverse experiences from 
the worlds of audiovisual productions, programming, and the web. 

https://2wa.isvery.ninja 


https://www. linkedin.com/in/leonvankammen 


LiSA Software by Frode Hegland, featuring the voice of Janine Earl. 


Discontinued: http://www. liquid.info/lisa/ 


Machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building 


methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. 
A type of Artificial Intelligence (AI). 
https://en. wikipedia. org/wiki/Machine_learning 


#Technology. Timeline: 20th century-. 


Map View in Author. 
Here the user can place text anywhere they want. 


If there are defined concepts on the map, the user can click on them and lines will emanate to any text 


on the map which is in that text’s definition. 


Matthias Miiller-Prove is an independent innovation and interaction designer. He is an experienced 


public speaker and author on the areas of innovation, interaction design, and media theory. 


https://mprove.de 


Matthias Miiller-Prove I'm a computer scientist, interaction designer and the founder of the Chrono 
Research Lab. So doing lots of heritage stuff. Also sometimes refer to myself as a kind of software 


archaeologist. That was my introduction. 


Matthias Miiller-Prove is an independent innovation and interaction designer. He is an experienced 


public speaker and author on the areas of innovation, interaction design, and media theory. 


https://mprove.de 


Meta is a large social media company which is focusing on what they term the ‘metaverse’. 
Formerly called Facebook. 
https://about.facebook.com 


Timeline: 4 February 2004- 


metadata information about other information, in the case of documents, this can include structural 
information (headings for example), how they connect to other documents (References) and who 


created the document (self-citing) 


METADATA is information about other information, in the case of documents, this can include 
structural information (headings for example), how they connect to other documents (References) and 


who created the document (self-citing) 


Metaverse From Wikipedia: “A metaverse is a network of 3D virtual worlds focused on social 


connection” 


“The term metaverse was coined in Neal Stephenson's 1992 science fiction novel Snow Crash” 


https://en. wikipedia. org/wiki/Metaverse 
At the time of writing, 2022, the term has been popularised by Meta, the company previously called 
Facebook. Most often used in conjunction with AR/VR augmented environments, not such much the 


3D worlds where the user access the world through a flat screen. 


Names View in Author and in Reader (with Visual-Meta) to show all named entities (and headings). 


NIC Doug Engelbart concept: Networked Improvement Community “Consider an "Improvement 
Community" (IC) as collectively engaged in improving an agreed-upon set either of individual 
capabilities, or of collective group capabilities-e.g. a professional society. Let's introduce a new 
category, a "Networked Improvement Community" (NIC): an IC that is consciously and effectively 
employing best-possible DKR (Dynamic Knowledge Repository) development and usage.” 


(augmenting society's collective IQ). 


OpenAL is an artificial intelligence (AI) research laboratory consisting of the for-profit corporation 


OpenAI LP and its parent company, the non-profit OpenAI Inc. 
Products: GPT-1. GPT-2. GPT-3 
https://openai.com 


https://en. wikipedia.org/wiki/OpenAI 


OpenAL is an artificial intelligence (AI) research laboratory. 

“consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit OpenAI 
Inc. 

The company, considered a competitor to DeepMind, conducts research in the field of AI with the 
stated goal of promoting and developing friendly AI in a way that benefits humanity as a whole. The 
organization was founded in San Francisco in late 2015 by Elon Musk, Sam Altman, and others, who 
collectively pledged US$1 billion. Musk resigned from the board in February 2018 but remained a 
donor. In 2019, OpenAI LP received a US$1 billion investment from Microsoft.” 

https://en. wikipedia.org/wiki/OpenAI 


https://openai.com 


Products include: GPT-3 and DALL-E. 


Founded by Elon Musk, Sam Altmanllya Sutskever, Greg Brockman, Wojciech Zaremba and John 


Schulman. 


Timeline: 11 December 2015-. 


Paper is a general term for a student or academic document in general. 
Primarily in PDF when published or handed in. 


In manuscipt/editable/personal form it is generall in the Microsoft Word format. 


Patrick Lichty Assistant Professor, Animation/Multimedia Design, College of Arts and Creative 


Enterprises, Zayed University, Abu Dhabi 


Patrick Lichty Assistant Professor, Animation/Multimedia Design, College of Arts and Creative 


Enterprises, Zayed University, Abu Dhabi 


PDA ‘Personal Digital Assistant’. This was before smartphones. 


PDF ‘Portable Digital Format’ developed by Adobe, now free with no license restrictions. It is a print 
to digial medium with few digital affoardances which my work on Visual-Meta expands to allow for 
users to interact with the document in useful ways, while staying compatible with the basic PDF 


format. 


It is an export format rather than a manuscript/working format. 


PDF ‘Portable Digital Format’ developed by Adobe, now free with no license restrictions. It is a print 
to digital medium with few digital affordances which my work on Visual-Meta expands upon to allow 
for users to interact with the document in useful ways, while staying compatible with the basic PDF 


format. 


perspectives how you look 


Phil Gooch Founder & CEO Scholarcy. 


https://www.scholarcy.com 


Phil Gooch Founder & CEO Scholarcy. 


https://www.scholarcy.com 


Reader is my minimalist PDF viewer for macOS, soon also for iOS, which can read any PDF and can 
provide added interactions if the PDF has Visual-Meta attached, which can either be produced by 
Author or any other word processor with Visual-Meta capability, or downloaded from an online 
repository which features Visual-Meta, such as the ACM digital library. 

It is produced by my small, independent software development company The Augmented Text 


Company LTD, with programming by Jacob Hazelgrove. 


https://www.augmentedtext.info 


Reader a PDF viewer from the Augmented Text company. 


https://www.augmentedtext.info/reader 


references is a list of all the citations a document uses, in an Appendix. In-Body citation, point to 
these References. This language is not fixed, it is sometimes used interchangeably with Bibliography 
but in my context a Bibliography is a list of work not expressly cited but which are relevant. 

in this context ‘Reference’ with uppercase ‘R’ refers to the appendix in an academic document which 
lists cites sources. In contrast, the citation in the body of the document is referred to as in-body 


citation. 


A form of metadata. 


robust is defined as the metadata being available for as long as the document is readable, even if the 


reader software changes or the data format of the document changes. 


Rollo Carpenter Artificial Intelligence Developer of Cleverbot. 


https://en.wikipedia.org/wiki/Rollo Carpenter 


Search Search in this context refers to instant search enabled by Liquid. 


Softspace “Softspace is inventing a new kind of tool for thought for thinkers and makers using virtual 


and augmented reality.” 
A tool for thought. 
https://soft.space 


Timeline: -. Keywords: VR. AR. 


Spatial Computing “was defined in 2003 by Simon Greenwold, as “human interaction with a 
machine in which the machine retains and manipulates referents to real objects and spaces”. With the 
advent of consumer VR, AR and mixed reality, companies use ‘spatial computing’ in reference to the 
practice of using physical actions (head and body movements, gestures, speech) as inputs for 
interactive digital media systems, with perceived 3D physical space as the canvas for video, audio, 


and haptic outputs. It is also tied to the concept of ‘digital twins’. 


https://en. wikipedia. org/wiki/Spatial_computing 


Speech to Text Analog Digital Conversion 


Structural inherent characteristics such as pages, headings etc. 


Student for my work, the student is the primary user of Author for writing and Reader for reading. 


Teacher in this context it the primary reader of a student paper. 


Ted Nelson, Theodor Holm Nelson coined the term ‘hypertext’. 


Text is the basic ’stuff’ of this work. 


The Future of Text A series of symposia and books under the name ‘The Future of Text’ produced by 


the same people who run The Augmented Text Company. 


https://futuretextpublishing.com 


The Future of Text, THE FUTURE OF TEXT is an initiative by Frode Hegland to promote 
dialogue around the potential of the future of text, or "Symbol Manipulation’ as Doug Engelbart called 


it more generally. The initiative includes: 
e The Future of Text Symposium since 2011. 
e The Future of Text Books 


e The Future of Text Journal 


Touch Touching a device to interact. 


UK https://en. wikipedia. org/wiki/United_Kingdom 


#Location 


USA https://en.wikipedia.org/wiki/United_States 


#Location 


ViewSpec Doug Engelbart concept: View Specifications for a user to see their work in different views 


to gain new perspectives and to navigate their document. 


ViewSpec Viewspecs — short for "view specifications”. Allows you to control your view of a 


document. 


https://www.dougengelbart.org/content/view/2 18/ 


Vint Cerf, Vinton Gray Cerf, Vinton G. Cerf Co-inventor of the Internet, VP and Chief Internet 
Evangelist for Google. Chairman of the Marconi Society. Former executive at MCI, the Corporation 
for National Research Initiatives, the Internet Corporation for Assigned Names and Numbers 
(ICANN), the American Registry for Internet Numbers (ARIN), the Association for Computing 
Machinery (ACM) and member of the Faculty of Stanford University. Fellow of IEEE, ACM, BCS, 
AAAS, American Academy of Arts and Sciences, and American Philosophical Society. Member of the 
US National Academies of Engineering and Science and foreign member of the Royal Society and the 


Royal Swedish Engineering Society. 


He has been a strong and vocal supporter of Frode Hegland’s work, and Visual-Meta in particular. 


Visual-Meta is my approach for embedding metadata in documents, initially PDF, to allow viewer 
software to provide rich interactions. The Visual-Meta is added as an appendix to the end of the 


document in a format inspired by BibTex. 


http://visual-meta.info 


Visual-Meta A open and robust way to augment flat PDF documents to make them more interactive, 


by The Augmented Text Company people. 


http://visual-meta.info 


Visual-Meta is Frode Hegland’s approach for embedding metadata in documents. 


Initially implemented in PDF, to allow viewer software to provide rich interactions. The Visual-Meta 
is added as an appendix to the end of the document in a format inspired by BibTex. Implemented in 


the Author software and the Reader software by The Augmented Text Company. 
http://visual-meta.info 


#Software | Timeline: - 


Voice Speaking and listening for interaction. 


VR ‘Virtual Reality’ is a simulated experience that employs pose tracking and 3D near-eye displays to 


give the user an immersive feel of a virtual world (Wikipedia). 


Included in the term XR. 


XR ‘eXtended Reality’, a term including both Virtual Reality (VR) and Augmented Reality (AR). 
This is a useful term since headsets increasingly feature passthrough video, allowing the user to toggle 


between VR and AR modes. 


#Technology | Timeline: 20th century- 


Endnotes 


@ https://www.dropbox.com/s/pnal fvfSsrbctweg/ 
xrOS%3A%20What%20to%20Expect%20From%20the%20Software%20Designed™%20for%20A pple 
%27s%20AR%3AVR%20HEeadset%20-%20MacRumors.pdf?dl=0 


P What performance-enhancing stimulants mean for economic growth 
https://www.economist.com/ 


https://www.economist.com/finance-and-economics/2023/05/25/what-performance-enhancing- 


stimulants-mean-for-economic-growth 
© Prepare for the Textpocalypse By Matthew Kirschenbaum in The Atlantic. 


https://www.theatlantic.com/technology/archive/2023/03/ai-chatgpt-writing-language-models/ 


673318/?utm_source=apple_ news 
d https://speechify.com 
e Apple ‘Live Text’ has gone some way to address this recently. 


https://support.apple.com/en-gb/HT2 12630 
f And some day, multiple high quiality projectors onto the world. 


E https://scholarlykitchen.sspnet.org/2023/04/13/textpocalypse-a-literary-scholar-eyes-the-grey-goo- 


of-ai/ 
h An article blogged and subsequently lost to hacking and spam. 


i Douglas Engelbart 1963: A Conceptual Framework for the Augmentation of Man’s Intellect. https:// 
dougengelbart.org/pubs/augment-133183-AHI-Vistas.html 


Ì Ivan Sutherland 1963: Sketchpad - A Man-Machine Graphical Communication System. https:// 
www.cl.cam.ac.uk/techreports/UCAM-CL-TR-574.pdf 


K Apple Lisa IN Matthias Miiller-Prove 2002: Vision and Reality of Hypertext and Graphical User 


Interfaces. https://mprove.de/Vvisionreality/text/3.1.8 lisa.html 
l Interview with Robert Cailliau, 2018. https://www.youtube.com/watch?v=9_B-ZGphJLk 


™ Matthias Miiller-Prove, Frank Ludolph 2007: Dueling Interaction Models of Personal-Computing 
and Web-Computing. https://mprove.de/script/07/medichi/paper.html 


n Matthias Miiller-Prove, 2011: Back to Childhood — Infantilisation of UI Design. https://mprove.de/ 


script/11/hyperkult/slides.html 

° Future of Text and AI, 20-Apr-2023: https://www.youtube.com/watch?v=qZfy_7WJwow 

P interview with M.Heilig, footage early 1980s. https://www.youtube.com/watch?v=vSINEBZNCks 
4 Ivan Sutherland. The Sword of Damocles. https://www.youtube.com/watch?v=NtwZXGprxag 


T Lana Wachowski and Lilly Wachowski 1999: The Matrix - stockroom video clip. https:// 
www.youtube.com/watch?v=UBK3DH0jTmQ 


S “Also heavily advertised in local newspapers and dailies That they are encouraging for people to 
come down and work And then we went to the embassy and they showed us Kew garden pictures and 
pictures of the various parts of England That it is all that beautiful and everything is just right And 
that's why we just applied for the voucher You burn my flame within your hands You know when my 
destiny falls This time has insecurity I feel makes me restless inside Will you take me there To a 
distant place I've never been before? I could leave this world I could follow you like oceans to the 
shore You could take me there Make the rivers of my mind flow to my dreams You hold your secrets 
from my eyes You see where the furthest rain falls The day breaks over in the streams You know 
where my rivers will flow Will you take me there To a distant place I've never been before? I could 
leave this world I could follow you like oceans to the shore You could take me there Make the rivers 
of my mind flow to my dreams And I dream of places far from here And I call your name to the wind 
And I wish the night would take me to another world Where no one knows a face or has a name Will 
you take me there To a distant place I've never been before? I could leave this world I could follow 
you like oceans to the shore You could take me there Make the rivers of my mind flow to my dreams 
Make the rivers flow Will you take me there? Will you take me there? Oh yeah Take me there Take 
me there Take me there yeah Take me there yeah yeah Take me there Will you take? Will you take me 
there? Take me there Take me there Take me there yeah Take me there yeah yeah Take me there Take 
me there Take me there yeah Take me there yeah yeah Make the rivers flow Make the rivers flow 
Make the rivers flow Make the rivers flow Make the rivers flow Make the rivers flow Make the rivers 


2 


flow 


https://www. youtube.com/watch?v=lIHv0oG9wPJc 


t The exception is there are specific names which can be rendered in text which provide the 


overviews. 
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Visual-Meta Appendix 


This is where your document comes alive. The information in very small type below allows software to provide rich 
interactions with this document. 
See Visual-Meta.info for more information. 


This is what we call Visual-Meta. It is an approach to add information about a document to the document itself on the same level of the content. The same as would be necessary on a physically printed page, as opposed to a data layer, since this data layer can be lost and it makes it harder for a user to take advantage of this data. Important notes are primarily about the 
encoding of the author information to allow people to cite this document. When listing the names of the authors, they should be in the format ‘last name’, a comma, followed by “first name" then ‘middle name’ whilst delimiting discrete authors with (‘and’) between author names, like this: Shakespeare, William and Engelbart, Douglas C. 4 Dates should be ISO 8601 
‘compliant. 4 The way reader software looks for Visual-Mta in a PDF is to parse it from the end of the document and look for @ {visualmeta-end}. If this is found, the software then looks for {@ {visual-meta-start} and uses the data found between these marker tags. I is very important to make clear that Visual-Meta is an approach more than a specific format and that it is 
based on wrappers. Anyone can make a custom wrapper for custom metadata and append it by specifying what it contains: For example @dublin-core or @rafs. 4 This was written Summer 2021. More information is available from https://visual-meta.info or from emailing frode@hegland.com for as long as we can maintain these domains. 

@{visual-meta-start} 


@{visual-meta-header-start} 


@yvisual-meta{ 
version = {1.1} generator = {Author 8.5.2 (1196)}.4} 


@tvisual-meta-header-end} 
@tvisual-meta-bibtex-self-citation-start} 


@anticle{2023-06-04T20:20:19Z/TheFutureo, 
author = {Frode Alexander Hegland) feditor = {Frode Alexander Hegland) title = {The Future of Text Special Edition 


“VR before Apple's Reality Strikes" 


4th of June 2023} ffilename = {VR before Apple's Reality Strikes:pdf} month = {jun} .fyear = {2023} finstitution = {University of Southampton} vm-id = {2023-06-04T20:20:192/TheFutureo} 4} 


@visual-meta-bibtex-sef-citation-end} 
@freferences-start} 


@nise{douglasCarlEngelbarv Augmenting, 
author = {Douglas Carl Engelbart) title= {Augmenting Human Intellect: A Conceptual Framework} publisher = {SRI} $year = {1962} Sur 
@book {peterGodfiey-Smith/OtherMinds, 

author = {Peter Godlrey-Smith} title = {Other Minds} publisher = (Farrar, Straus and Giroux} fyear = {2017} month = {10} furl = {htt:/playgoogle-com/books/reader?id-WhQ7VZAACAAI&hI-Rsource 
@book fandrewPettegree/TheBookint, 

author = {Andrew Pettegree} title= {The Book in the Renaissance} year = {2010} furl 
@book annieMurphyPaul/TheExtende, 

author = {Annie Murphy Paul} ti 
@book {martineArkady/AMemoryCal, 

author = {Martine Arkady} title = {A Memory Called Empire} fpublisher = {Pan Macmillan} year = {2019} fmonth = (4) furl 
@book vingeVernor/Rainbow'sE, 

author = {Vinge Vernor title= {Rainbow's End} publisher = {Pan Macmillan} year = {2011} month = {12} furi 
@book {mattWilkinsowRestlessCi 

author = {Matt Wilkinson} title = {Restless Creatures} {publisher = {Icon Books Ltd} $year = {2016} month = {5} ul = 
@mise{10.1145/191666.191712, 

author = {Bruce Tognazzini} title = {The “Starfire” video prototype project}.fjoural = {Proceedings of the SIGCHI conference on Human factors in computing systems celebrating interdependence - CHI 94) publisher = {ACM Press} $year = {2003} month = {11} address = {New York, New York, USA} furl = {htp/dx doi. orp/10.1145/191666.191712} fdoi = 
{10.1145/191666.191712} 4} 

@book {martineArkady/AMemoryCal, 

author = {Martine Arkady} title= {A Memory Called Empire} fpublisher = {Pan Macmillan} year = {2019} fmonth = (4) furl 
@book {martineArkady/AMemoryCal, 

author = {Martine Arkady} ftitle = {A Memory Called Empire} $publisher = {Pan Macmillan} year = (2019},fimonth = (43 Sur 
@book {murrayLeinster/ALogicName, 

author = {Murray Leinster} title = {A Logic Named Joe} publisher = {Baen Books} year 
‘@book {murrayLeinster/ALogicName, 

author = {Murray Leinster} title = {A Logic Named Joe} publisher = {Baen Books} year 
@mise tomislavBezmalinovie/MetaQuest3, 

author = {Tomislav Bezmalinovie} title= {Meta Quest 3 has been officially announced: This is what the new headset brings} publication 
@mise {macRumorsStall/Apple'sARY, 

author = {MacRumors Staff} $title = {Apple's AR/VR ‘Reality Pro' Headset} fpublication = {MacRumors} {publisher = {MacRumors.com, LLC. } {year = {2023},fmonth = {6} furl = {htips:/www.macrumors.com/rounduplar-vi}} 

@mise {jonWabbeking/WillSamsun, 

author = {Jon Webbeking} tle = {Will Samsung unveil its new VR headset before the end of 2023} {publication = {MIXED Reality News } $publisher = {DEEP CONTENT GBR} year = {2023} month = {5} furi = {https/imixed-news.com/en/wil-samsung-unveil-its-new-vr-headset-before-the-end-of:2023/} §} 


= {hups:/www-dougengelbart ong/pubs/augment-3906. uml} } 


tbs_api} isbn = {0374537194, 9780374537197) f} 


= {http:/play-google.com/books/reader?id-BC4CHAAACAAI&hI=&source=gbs_api} isbn = {030011009X, 9780300110098} 4} 


k- DWAAQBAJ&hl=&source=gbs_api},fisbn = (9780544947665, 0544947665} 4) 


{The Extended Mind} publisher = (Eamon Dolan Books} year = {2021} furl = {hitp:/iplay-google.comvbooks reader? 


{Iutp:/play google com/books/reader?id-p7RzDwAAQBAJ&hI=&source=gbs_api} isbn = {9781529001600, 1529001609} 4} 


= {http:/play-google.com/books/reader?id~AiEscCXu7TIC&hI=8source=ghs_api} isbn = (9781447217473, 1447217470} $} 


{htp://play google comvbooksireader?id=zdw7CwAAQBAJ&hl-&source=gbs_api} isbn = (9781785780462, 1785780468}.4} 


{Iutp:/play google com/books/reader?id-p7RzDwAAQBAJ&hI=&source=gbs_api} isbn = {9781529001600, 1529001609} 4} 


{utp://play.google.com/books/reader?id-p7R2DwAAQBAJ&hI=&source=gbs_api} {isbn = {9781529001600, 1529001609} 4} 


= {2005}, url = ¢hutp://play-google.com/books/teaderid=~CZGV6hSIQzIC&hI=Rsource=gbs_api} fisbn = (9780743499101, 0743499107} $} 


= {2005}, url = {http://play.google.com/books/reader?id-CZGV6h5fQzIC&hl=&source=gbs_api) fisbn = {9780743499101, 0743499107) $} 


(MIXED Reality News | publisher = {DEEP CONTENT GBR} year = 


2023} month = {6} furl = {https://mixed-news.com/en/meta-quest-3-announcement {} 


@mise flukeðones/MicrosofP, 
author = {Luke Jones} title = {Microsoft Patent Points to Upcoming Xbox VR Gaming Headset} publication = {WinBuzzer} -year = {2023} fmonth = 
@book fbenediktMichael/Cyberspace, 

author = {Benedikt Michael) tile = {Cyberspace} publisher = {Mit Press} year = {1992} furl = {http://play.-google.com/books/reader?id= XINQgAACAAJ&hI=&source= 
@mise {/Homețsines, 

title = {Home | sinespace} publisher = {Sine Wave Entertainment Ltd} fur! = fhtips://sine.space}fquote = {SineSpace) f} 

@mise fdougEngelbart/Viewspecs—, 


{5} furl = {https://winbuzzer.com/2023/05/02/microsoft-patent-points-to-upcoming-xbox-vr-gaming-headset-xexwbn/} 1|} 


bs_api} isbn = (0262521776, 9780262521772) 1} 


author = {Doug Engelbart} title = {Viewspecs — Doug Engelbart Institute} url= {htps:/dougengelbart.org/content view/218/)} quote iD 
@misc [/XRFragment, 
title = {XR Fragments} year = {2023} fmonth = {5} furl = {htips://xrf'isveryninja#XR420Fragments} quote = {XR Fragments} §} 


‘@book tjefRaskin/TheHumanel, 
author = {Jef Raskin} title = (The Humane Interface} publisher = {Addison-Wesley Professional} {year = {2000} furi = (http:/play-google.com/books/reader?id-D39yjmL-f03kC&hI-& sour 
@misc{10.4018/978-1-60960-567-4.ch29, 

author = {Peter J. Wasilko} title = {Law, Architecture, Gameplay, and Marketing} {publication = {Business, Technological, and Social Dimensions of Computer Games} publisher = {IGI Global} {year = {2011} month = {1} url = {http://dx.doi.org/10.4018/978-1-60960-567-4} foi = {10.4018/978-1-60960-567-4.ch29} isbn = {9781609605674} pages = 
{432-447} fpageRange = {432-447}.4} 

@book {nealStephenson/SnowCrash, 

author = {Neal Stephenson} title = {Snow Crash} publisher = {Penguin UK} qyear = {1994} month = {10} furl = {http:/play google.com/books/reader?id~inYs79gV4UQC&hI-&source=gbs_api} fisbn = (9780141924045, 0141924047} 4} 

@mise{10.1145/263407 26350: 
author = {Jun Rekimoto} ttle = {Pick-and-drop} $joumal = {Proceedings of the 10th annual ACM symposium on User interface software and technology - UIST'97} publisher = {ACM Press} fyear = {1997} month = {10} faddress = {New York, New York, USA} furl = {http://dx.doi.org/10.1145/263407.26350S} doi = {10.1145/263407.263505} 4) 
@misc david Small/Rethinking, 

author = {David L. Small} title 
@mise{10.1145/259963.260437, 
author = {David Small and Suguru Ishizaki and Muriel Cooper} title = {Typographic space} journal = {Conference companion on Human factors in computing systems - CHI 94} publisher = {ACM Press} fyear = {2003} {month = {11} address = {New York, New York, USA} url 
@mise{10.1145/259081.259: 
author = {David Small} ttle = {Hand held tools for navigating information} journal = {ACM SIGGRAPH 97 Visual Proceedings: The art and interdisciplinary programs of SIGGRAPH '97 on. - SIG 
{10.1145/259081,259225}.4} 
@misc{10.1145/1120212.1120437, 

author = {David Small and Hiroshi Ishi} th 
{10.1145/1120212.1120437}.$} 

@book {chinaMigvillerTheCity&Th, 

author = {China Migville} title = {The City & The City} publisher = {Pan Macmillan} year = {2018} month = {7} furl = {hitp:/play.google.com/books/teader?id=PKRIDwAAQBAJ&hI-8source=gbs_api} fisbn = (9781529010268, 1529010268} 4} 
@misc leahCrane/Itmaybepos, 

author = {Leah Crane} title = {It may be possible to traverse a wormhole and then send a signal home} journal 
@mise{frodeA lexanderHlegland/FabienBene, 

author = {Frode Alexander Hegland} title = {Fabien Benetou & Vint Cerf at The Future of Text 2022} year = {2022} month 


bs_api} isbn = (0201379376, 9780201379372) 1} 


{Rethinking The Book} {publisher = {MIT Media Lab} year = {1999} month = {1} furl = {https//acg.media.mit.edu/projects/thesi/DSThesis pd quote = {Rethinking the Book} 4} 


{Http:/dx. doi org/10.1145/259963.260437} doi = {10.1145/259963.260437},4} 


RAPH '97} publisher = {ACM Press} year = {2005} month = {10} address = {New York, New York, USA) furl = {http:/dx doi.org/10.1145/259081.259225} doi = 


{Design of spatially aware graspable displays} {journal = {CHI '97 extended abstracts on Human factors in computing systems looking to the future - CHI'97} publisher = {ACM Press} year = {2006} fmonth = {5} address = {New York, New York, USA} furl = {hitp://dx doi org/10.1145/1120212.1120437} doi 


12} furl 


New Scientist} {year = {20} month https:/www.newscientist.comvarticle/2352648-it-may-be-possible-to-traverse-a-wormhole-and-then-send-a-signal-home!} 4} 


GETI 
@references-end} 
@{alossary-start} 


Gentry 
name= {academic} description = {someone who reads academic documents, usually associated with an academic institution, but not necessarily },§} 

@entry| 

name = {ACM} description = {From Wikipedia: “The Association for Computing Machinery (ACM) is a US-based international learned society for computing. It was founded in 1947 and is the world's largest scientific and educational computing society.” 

hutps:/en.wikipedia.org/wiki/Assoctation_for Computing Machinery 

ACM's ‘Hypertext’ Conference piloted Visual-Meta in 2021.31} 

Gentry 

name = {Addessabilty} {description = {refers to how something can refer to something else. In the digital word this primarily means in a local file structure and a server based network. In academia is can refer to citation or references. In the physical world it can refer to a formal location layout, such as a flat so and so in house so and so, as well as co-ordinates. It can also 


man relative addressing, such as saying take a left after the third yellow house. }.4} 


@ontry| 
name = {AGI} description = {(Artificial General Intelligence). AI stage 2. 

If we are to see a world like the movies with artificial intelligence, AGI will need to come into full fruition, AGI would mean a machine would be capable of understanding the world as well as any human. All while maintaining the same capacity to leam how to carry out a huge range of tasks. The reality? AGI is not yet among us. It is a hypothetical form of AL. The 
potential is there but some believe it to be decades away! 

‘According to a study by Muller and Vincent C., the top-cited 100 AT researchers have a collective 90% confidence that AGI will be among us by 2070. To determine if machine is truly AGI, it has to pass the tests: 

- Turing Test 

In this test, 2 humans and a machine look at something they haven't seen and converse about it. The conversation is recorded and an evaluator determines which descriptor is the machine. The machine is only disqualified if the evaluator is positive it can identify it. (check out Touring Test on Wikipedia ) 

- The Coffee Test (Wozniak) 

If AGI machines need to understand the world as well as humans do, they must know how to make a good coffe. In this test an ai robot is required to go into an average home and figure out how to make coffee. It involves finding the coffee machine and adding coffee, as well as sourcing a mug and correctly brewing the hot drink. 

- Robot College Student Test 

‘A literal test! the machine would take the same university classes as humans and complete the exam. Some may argue that the machine is cheating with its built-in search engines but who is going to argue with a robot. 

- Employment Test 

This is where we should begin to worry about an I-Robot world. This test determines whether a machine can perform at last as well as humans in the same jobs!) 1} 

Gentry 


\tps://en,wikipedia.org/wiki/Artficial_intelligence 


AL} description = {*Artficial intelligence” is 
‘Technology | Timeline: 20th century-}.4} 

@entry | 

name= {ANI} description = {(Astificial Narrow Intelligence) 
AI Stage 1 


intelligence demonstrated by machines, as opposed to the natural intelligence displayed by animals including humans.” 


‘Often called Weak AI, this is the AI already embedded in our daily lives. A nice way to describe it is that ANI is super-human at specific singular tasks. Like teaching an AI machine to play chess! and it will master it beyond human potential with incredible consistency. Challenge it in draughts though? and you'll get a much needed ego boost. This machine can only play 
chess. Without prior training and programming in draughts, it doesn’t stand a chance. Think of AI as hundreds of interns capable of highly specific tasks more efficiently than humans. 


ANI has already transformed the way we live, but it has its limitations. A lot of you will already interact with ANI on a daily basis. Think of talking with Siri or Alexa on your smart devices, or facial recognition when opening your phone! 


i 


@entry 

name = {Annie Murphy Paul} {description = {From an interview in GQ: “We make better use of our cognitive resources, says Paul, when we use them in conjunction with “extra-neural” resources: our body (embodied cognition), our environment (situated cognition), and the people around us (distributed cognition) The brain evolved to move the body, to navigate through 
space, to interact with other people,” says Paul. 

“Those are these human strengths that we're totally putting aside when we focus on the brain and we think, “To get real thinking and real work done, I have to sit still, not talk to anybody, and just push my brain harder and harder.’ It’ just not working very well.” 

hitps:/‘vww:gq.convstory/extended-mind-annie-murphy-paul 

Author of The Extended Mind. 

Timeline: -.}.4} 

@entry 

name = {AR} {description = {*Augmented Reality’ is an interactive experience that combines the real world and computer-generated content, The content can span multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. (Wikipedia). 

Included in the term XR. } 4} 

@entry| 

name = {AR} description = {includes the real reality, as opposed to VR which does not. 

“Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.” 

hitps:/en.wikipedia.org/wiki/ Augmented reality 

‘Technology | Timeline: 20th century-}} 

@entry 

name = {AR}, description = ("Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.” 
hitps:(éen.wikipedia.orgiwiki/Augmented reality 


AR includes the real reality, as opposed to VR which does not.) §} 
@entry 

name = {ARC} description = {Doug Engelbart Organisation: Augmentation Research Center, The name of Doug's lab at SRI where he proposed a system called H-LAM/T in 1962 and developed and in 1968 demonstrated NLS: oNLine System, his platform for shared knowledge work research, later renamed Augment and from which I decided on the name Author, since 
‘Author and Augment share etymological roots.}.} 

@entry 

name = {Article} description = {A published paper, in a Journal, for academic workflows. |) 

@entry| 

name = {Artificial General Intelligence} {description = {Stage 5- 


(AGI This is the main objective of many scientists working in the field of AT- to develop a machine with intelligence like humans. The artificial general intelligence or self-aware systems are not in application to date but not many years left to discover this avenue. This is the stage represented in many scifiction movies that machines are leading humans in intelligence.}-} 
@entry{ 


ASI} fdeseriptior 
Stage 3 AL 


{Artificial Super Intelligence) 


Just when you think AI has reached its limit, we bring you the AT of all Al. Super-Intelligence is quite self-explanatory. Oxford Philosopher Nick Bostrom describes ASI as “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest”. Again, this is a hypothetical agent and one which is decades away. 
ASI has huge potential but it also has huge risk. Stephen Hawkings feared its development could be the worst event in the history of our civilisation. If we create superintelligent machines we may put our own existence at risk, but this is just the plot for another sci-fi film. 


The reality is that AI is a tool for humans to better our existence. Advancements in artificial intelligence mean the skys the limit! The question is — will the limit bring more positives or negatives? We must tread carefully with AT whilst simultaneously embracing i} 4} 
@entry{ 

name = {Augmented Text Company} {description = {The company which produces these tools: Author, Reader and Liquid. 

Director is Frode Hegland. 

https://www augmentedtext info} 4) 

@entry{ 

name = {Author} description = {means in the context of this work someone who writes a document, which includes academic papers and books. 

Etymology: “mid-14c., auctor, autour, autor “father, creator, one who brings about, one who makes or creates" someone or something, from Old French auctor, acteor “author, originator, creator, instigator" (12e., Modern French auteur) and directly from Latin auctor “promoter, producer, father, progenitor; builder, founder; trustworthy writer, authority; historian; performer, 
doer; responsible person, teacher,” literally “one who causes to grow,” agent noun from auctus, past participle of augere “to increase,” ftom PIE root *aug- (1) “to increase. ™ From https://www.etymonline.com/search?q=Author 

Category: Software 

Timeline: - 


Can also refer to the Author software by The Augmented Text Company. 
1 

@entry 

name = {Author} {description = {is my word processor for macOS, maybe later for iOS, which produces PDFs with Visual-Meta for any PDF viewer but which my Reader PDF viewer can parse to give the user rich interactions. It is produced by my small, independent software development company The Augmented Text Company LTD, with programming by Jacob 
Hazelgrove. 

hitps:/vwwaugmentedtext.info}.4} 

@entry| 

name = {Barbara Tversky} {description = {Guest presenter at the Future Text Lab, 

Issue 1.1 on the 21st of January 2022. 

Professor emerita of psychology at Stanford University and a professor of psychology and education at Teachers College, Columbia University. Tversky specialises in cognitive psychology. 

Author of Mind in Motion: How Action Shapes Thought. Basic Books, 2019.}.4} 

@entry 

name = {Benediktine Cyberspace} {description = {A 3-D visualization inspired by Michael Benedikt's seminal text, “Cyberspace: First Steps} f} 

@entry| 

name = {Book} description = {a published work, on any substrate inclduing paper or digital. 

Generally longer than a ‘paper’ but in terms of digital work, there is litle technical difference, the difference is mostly social.) 9} 

@entry 

name = {Bootstrapping} description = {Doug Engelbart concept.: The act of co-evolving the tool and human systems to make better tools and systems, thus pulling us up by our bootstraps. (For an overview, see the Bootstrap Paradigm Map). This is done on three levels of activity.) §} 

@entry| 

name = {Christopher Gutteridge}. description 
Lives: Southampton, UK. 

Works at the University of Southampton} §} 
@entry 

name = {Citations} qdescription = {In this paper, the term’ citation’ is both the ‘citation’ mention in the body of the document, and its corresponding ‘references’ entry for the source material in the Reference Appendix at the end of the document, though for the sake of clarity I will mainly use ‘citation’ to refer to the in-body half and ‘reference’ for the entry ofthe source in 
the References section 


is also a contributor to the first volume of The Future of Text. 


inl 

@entry{ 

name = {Cognition} description = {*is the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses”. 
hutps:/en.wikipedia.org/wikCognition} A} 

@entry{ 


name = {concept} alt-namel = {defined concept} falt-name2 = {defined concepts} {description = {Means in this context, my experiment with useful units of thought or knowledge, which can be defined in Author, hopefully soon in Reader. Stored as text. 
The Map view uses the definition to draw lines where text from a definition is also present on the Map. 

When a document is exported to PDF the Defined Concepts become Glossary Terms. 

This is as opposed to inferred concept. 

‘Also aView in Author and in Reader (with Visual-Meta) to show all named entities (and headings).)} 

@entry{ 

name = {Cyberspace} description = {The term 
@entry{ 

name = {David Johnson} ‘description = (Co-author of Law and Borders, The Rise of Law in Cyberspace} 4} 

@entry| 

name = {Deena Larsen} description = {is a new media and hypertext author involved in the creative electronic writing community since the 1980s. From USA. 
hutps:/en.wikipedia.org/wiki/Deena_Larsen 


-yberspace” first appeared in the visual arts in the late 1960s, when Danish artist Susanne Ussing (1940-1998) and her partner architect Carsten Hoff (b. 1934) constituted themselves as Atelier Cyberspace.” Used in this context to mean a digital world, not necessarily in VR or AR.} 4} 


Timeline: 1964-}.4} 
Gentry| 

name = {Definition} description = {when it comes to glossaries, it is the user's stated meaning ofa term.) §} 
Gentry, 

name = {Document} fdeseription = {Can be any type of encapsulated information on a computer.) } 
@entry | 


name = {DOI} description = {Document Object Identifiers. An effort to make addressing academic documents via the web more robust. 
Used in Author to let the user paste a DOI to cite an academic document which is then sent to CrossRef to be parsed into BibTeX which is then used to create a full citation. }-f} 
@enryt 


Doug Engelbart} falt-namel = {Douglas Carl Engelbart} falt-name2 = {Doug} {description = (“He was an engineer and inventor, and an early computer and Internet pioneer. He is best known for his work on founding the field of human-computer interaction, particularly while at his Augmentation Research Center Lab in SRI Intemational, which resulted in 
creation ofthe computer mouse, and the development of hypertext, networked computers, and precursors to graphical user interfaces. These were demonstrated at The Mother of All Demos in 1968. Engelbar’ law, the observation that the intrinsic rate of human performance is exponential, is named after him.” 

hutps:/en.wikipedia.org/wiki/Douglas Engelbart 

He was also my mentor and greatly influenced my work, resulting in my company called The Augmented Text Company and my word processor being called Author, in honour of his ‘Augment’ system. Visual-Meta is inspired by his Open Hyperdocument work }} 

@entry| 

name= {Doug Engelbart} description = (“He was an engineer and inventor, and an early computer and Intemet pioneer. He is best known for his work on founding the field of human-computer interaction, particularly while at his Augmentation Research Center Lab in SRI Intemational, which resulted in creation of the computer mouse, and the development of hypertext, 
networked computers, and precursors to graphical user interfaces. These were demonstrated at The Mother of All Demos in 1968. Engelbarts law, the observation that the intrinsic rate of human performance is exponential, is named after him.” 

huups:/en.wikipedia.org/wiki/Douglas Engelbart 

He was also my mentor and greatly influened my work, resulting in my company called The Augmented Text Compant and my word processor being called Author, in honour of his “Augment” system, Visual-Meta is inspired by his Open Hyperdocument work.} 


@entry 

name = {Egypt} {description = {https:/en.wikipedia.org/wiki/Egypt 

#Location} M} 

@entry | 

name = {Fabien Benetou} {description = {Prototyping - European Parliament Innovation lab WebXR consultant - Former UNICEF Innovation Fund WebXR technical advisor. 


hitps:/twitter.com/utopiah 
hitps:/fabien.benetou.fr} 4} 

@entry| 

name = {Find} {description = {View in Author and in Reader (with Visual-Meta) where the user can see only the senteces with keyword} §} 

Gentry | 

name = {flatland} description = {a semi-humorous term we use to refer to traditional displays, as opposed to the augmented environments of VR and AR. We fee! that it is important to be able to move data, including metadata between these environments. 4} 
@entry{ 

name = {Folded} Adescriptios 
Gentry | 

name = {Folding} description = {Folding of a document into a table of contents is enabled through Visual-Meta,} 4) 

Gentry | 

name = {Frode Hegland} {description = {Director of the Future Text Lab (FTL) https:/futuretextlab. info 

Director of The Augmented Text Company where he designed the macOS Author word processor, Reader PDF viewer and the Liquid text interaction tool. 

hitps:/vwwaugmentedtext.info! 

Editor of the "The Future of Text series of books and Journal, which will be the basis of future "The Future of Text volumes. 

Hosts 'The Future of Text Symposium’ annually, starting 2011 

Designed Visual-Meta, 

His mentor was Doug Engelbart and influenced by Ted Nelson.) 

@entry{ 

name = {Future Text Lab} {description = {A lab dedicated to studying how working with knowledge, particularly text, can be done effectively in VR/AR/Metaverse, We host Guest presenters and publish a Joumal which is part of The Future of Text series. 
htps:/futuretextlab info} 


{View in Author and in Reader (with Visual-Meta) where the user ean fold the document into an outline.) 4} 


@entry{ 

name = {Gesture} description = {Interaction through hands and body.) 

@entry{ 

name = {glossary} description = {A glossary in this context means a local definition, by the author or editor of the document.} 4} 
Genty 


name = {Glossary} description = {means, in the context of this work, terms and definitions in a ‘Glossary’ Appendix in a document. 
This can be created in Author as a Defined Term which is then exported as a Glossary and included in Visual-Meta. 

Reader can parse the glossary terms in the Visual-Meta and make them interactive. 

This is different from a dictionary since dictionary definitions have general validity and Glossary terms only aim to express the author’s point of view. 
Defined Concept becomes Glossary Term when exported from Author to PDF. 

Inspired by discussions with Doug Engelbart 

A form of metadata. 

A 

@enry{ 


Glossary {description = {In the context of my work and thinking, a glossary is a set of Defined Concepts which an author has created and which is then exported as a Glossary for the reader. 
The primary purpose of this is for the author to have to think through their writing by explicitly stating what something is. The definition of the defined concept can then include text which also has a definition and in the Map view in Author this connection can be shown as a line when either defined term is selected, 


The secondary purpose is for the author to edit this Glossary to make sure it is coherent for export and it can then help a reader understand the author's intentions.) §) 
Gentry} 


{Google} description = {is an American multinational technology company that focuses on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics." hnips:/en.wikipedia.org/wiki/Google 
Timeline: 1998- Location: USA} 4} 
@entry{ 


GPT} $deseription 
hitps:/en.wikipedia.orgiwiki/GPT 
hutps:/éen.wikipedia.orgiwikiJGPT-3 

Product of OpenAL.}§} 

@entry | 

name = {HMD} description = {Head Mounted Display for use in VR or AR.} f} 

@entry 

name = {HTML} {description = {"The HyperText Markup Language or HTML is the standard markup language for documents designed to be displayed in a web browser.” hitps//en. wikipedia org/wiki/HTML 
technology we are also using, along with PDF} §} 


{Generative Pre-trained Transformer” inelduinig GPT-2 and GPT-3. 


@entry{ 

name = {hypertext} description = {a term invented by Ted Nelson for interactive and connected digital text, Today it is very much about clicking on links but in Ted's early work it was all about freedom for the use.) 4} 
Gentry, 

name = {Hypertext} description = fa term invented by Ted Nelson for interactive and connected digital text, Today it is very much about clicking on links but in Ted's early work it was all about freedom for the user} ) 
@entry{ 


name = Immersed} {description = {Immersed is a virtual reality product, working productivity software for virtual offices 
hutps://immersed.com1 1} 

@entry{ 

name = {Intelligence Augmentation} description = {Doug Engelbart presented the notion of augmenting human intellect in his seminal 1962 paper, which would be the basis for his ‘mother of all demos” in 1968 


“We need to improve how we augment a group's (small, large, internal, global etc.) capability to approach urgent, complex problems to gain more rapid and better comprehension (which can be defined as more thorough and more critical) which result in speedier and better solutions (more contextual, longer lasting, cheaper, more equitable etc.) And furthermore, we must 
improve our improvement process (as individuals and groups).” 
Doug Engelbart (Engelbart, 1962) 


ih 

@entry{ 

name= {Journal} description = {A collection of articles. For academic use, usually the record of the work of society, such as any of the many special interest groups at ACM. |} 
@entry{ 

name = {Journal} {description = {The monthly Joumal of the Future Text Lab. Inspired by Doug Engelbart’s journal and academic joumals in general. Published as PDF with Visual-Meta and in liquid and soon HTML on https://futuretextpublishing.com/journal/} 4} 
Geny 

name = {Leon van Kamme} ‘description = {Leon is a Dutch audiovisual prototyper and researcher currently based in Budapest, Hungary. 

He is currently working on XR Fragments, in which he strives to bridge his diverse experiences from the worlds of audiovisual productions, programming, and the web. 
hutps:/2waLisvery.ninja 

hugps:/wvwwlinkedin.convin/Teonvankammen} 4) 

@entry{ 

name = {LiSA} description = {Software by Frode Hegland, featuring the voice of Janine Earl. 


Discontinued: http://www.liquid-info/lisa/} f} 


@enry 
name = {Machine leaming} {description = {Machine leaming (ML) is a field of inquiry devoted to understanding and building methods that Tear, that is, methods that leverage data to improve performance on some set of tasks. 
A type of Artificial Intelligence (AI) 


hugps:/en.wikipedia.org/wiki/Machine learning 
‘Technology. Timeline: 20th century} 

@entry| 

name = {Map} {description = {View in Author. 

Here the user can place text anywhere they want 

If there are defined concepts on the map, the user can click on them and lines will emanate to any text on the map which is in that texts dfinition.| 4} 
@entry{ 


Matthias Maller-Prove} description = fis an independent innovation and interaction designer. He is an experienced public speaker and author on the areas of innovation, interaction design, and media theory. 
higps:improve.de} M) 

Gentry, 

name= {Matthias Mller-Prove} description = {I'm a computer scientist, interaction designer and the founder of the Chrono Research Lab. So doing lots of heritage stuff: Also sometimes refer to myself as a kind of software archaeologist. That was my introduction. |} 
@entry | 

name = {Matthias Miller-Prove} description = {is an independent innovation and interaction designer. He is an experienced public speaker and author on the areas of innovation, interaction design, and media theory. 

hugps:improvede} 4} 

@entry{ 

name = {Meta} description = {is a large social media company which is focusing on what they term the ‘metaverse’. 

Formerly called Facebook. 

hutps:/aboutfacebook.com 

Timeline: 4 February 2004-} 4} 


@entry 

name = {metadata} description = {information about other information, in the case of documents, this can include structural information (headings for example), how they connect to other documents (References) and who created the document (self-iting)} 4} 
@entry| 

name = {METADATA} description = {is information about other information, in the case of documents, this can include structural information (headings for example), how they connect to other documents (References) and who created the document (selfciting)} 4} 
@entry 


name = {Metaverse} description = {From Wikipedia: “A metaverse is a network of 3D virtual worlds focused on social connection" 
“The term metaverse was coined in Neal Stephenson's 1992 science fiction novel Snow Crash” 

hutps:/en.wikipedia.org/wiki/Metaverse 

‘At the time of writing, 2022, the term has been popularised by Meta, the company previously called Facebook. Most often used in conjunction with AR/VR augmented environments, not such much the 3D worlds where the user access the world through a flat screen. 4} 
Gentry | 

name = {Names} description = {View in Author and in Reader (with Visual-Meta) to show all named entities (and headings).} 9} 

Gentry 

name = {NIC} {description = {Doug Engelbart concept: Networked Improvement Community “Consider an "Improvement Community” (IC) as collectively engaged in improving an agreed-upon set either of individual capabilities, or of collective group capabilities-e.g. a professional society. Let's introduce a new category, a "Networked Improvement Community" (NIC): 
an IC that is consciously and effectively employing best-possible DKR (Dynamic Knowledge Repository) development and usage.” 

{augmenting society's collective 1Q).}.f} 

Gentry 

name = {OpenAl} description = {is an artificial intelligence (AI) research laboratory consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit OpenAL Ine. 

Products: GPT-1, GPT-2. GPT-3 

hutps:/opensi.com 

hutps:/en.wikipedia.org/wiki/OpenAL}} 

@entry| 

name = {OpenAl}4description = {is an artificial intelligence (AI) research laboratory. 

“...consisting ofthe for-profit corporation Open AI LP and its parent company, the non-profit OpenA Inc. 

The company, considered a competitor to DeepMind, conducts research in the field of AL with the stated goal of promoting and developing friendly AT in a way that benefits humanity as a whole, The organization was founded in San Francisco in late 2015 by Elon Musk, Sam Altman, and others, who collectively pledged US$! billion, Musk resigned from the board in 
February 2018 but remained a donor: In 2019, OpenAL LP received a USS! billion investment from Microsoft.” 

hugps:/en.wikipedia.org/wiki/OpenAL 

hugps:/opensi.com 

Products include: GPT-3 and DALL-E. 

Founded by Elon Musk, Sam Altmanllya Sutskever, Greg Brockman, Wojciech Zaremba and John Schulman, 

Timeline: 11 December 2015-.}.4} 

@entry{ 

name = {Paper} description = fis a general term fora student or academic document in general 

Primarily in PDF when published or handed in. 

In manuseipteditabe'personal form it is general in the Microsoft Word format.) 

@entry| 

name = {Patrick Lichty} qdescription = {Assistant Professor, Animation/Multimedia Design, College of Arts and Creative Enterprises, Zayed University, Abu Dhabi}.4} 


Gentry 
name = {Patrick Lichty} description = {Assistant Professor, Animation/Multimedia Design, College of Arts and Creative Enterprises, Zayed University, Abu Dhabi} 4) 
@entry{ 

name = {PDA} description = {'Personal Digital Assistant’. This was before smartphones.) 

@entry{ 


name = {PDF} {description = (“Portable Digital Format’ developed by Adobe, now free with no license restrictions. It is a print to digial medium with few digital affoardances which my work on Visual-Meta expands to allow for users to interact with the document in useful ways, while staying compatible with the basic PDF format. 
It is an export format rather than a manuseript/working format.)} 


@entry 

name = {PDF} {description = {*Portable Digital Format’ developed by Adobe, now free with no license restrictions. It is a print to digital medium with few digital affordances which my work on Visual-Meta expands upon to allow for users to interact with the document in useful ways, while staying compatible with the basic PDF format.) §} 
Gentry | 

name = {perspectives} {description = {how you look} 4) 

Gentry | 

name = {Phil Gooch} description = {Founder & CEO Scholarey. 


hugps:/www:seholarey.com} A} 
@entry{ 

name = {Phil Gooch} description 
hutps:/www:seholarey.com} 4} 
@entry| 

name = {Reader} description = {is my minimalist PDF viewer for macOS, soon also for iOS, which can read any PDF and can provide added interactions if the PDF has Visual-Meta attached, which can ether be produced by Author or any other word processor with Visual-Meta capability, or downloaded from an online repository which features Visual-Meta, such as the 
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Registry for Internet Numbers (ARIN), the Association for Computing Machinery (ACM) and member of the Faculty of Stanford University. Fellow of IEEE, ACM, BCS, AAAS, American Academy of Arts and Sciences, and American Philosophical Society. Member of the US National Academies of Engineering and Science and foreign member of the Royal Society and 
the Royal Swedish Engineering Society. 
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isan initiative by Frode Hegland to promote dialogue around the potential of the future of text, or "Symbol Manipulation’ as Doug Engelbart called it more generally. The initiative includes: 
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{Virtual Reality’ is a simulated experience that employs pose tracking and 3D near-eye displays to give the user an immersive feel of a virtual world (Wikipedia). 
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